Sep 16 18:44:29.629904 ip-10-0-129-152 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:44:30.050614 ip-10-0-129-152 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:30.050614 ip-10-0-129-152 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:44:30.050614 ip-10-0-129-152 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:30.050614 ip-10-0-129-152 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:44:30.050614 ip-10-0-129-152 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:30.053294 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.053205 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:44:30.059344 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059329 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.059344 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059343 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.059344 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059347 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059350 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059353 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059356 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059359 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059362 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059365 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059368 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059371 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059374 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059377 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059379 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059382 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059385 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059388 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059391 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059394 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059396 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059399 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.059439 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059402 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059404 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059407 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059410 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059412 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059415 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059418 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059421 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059424 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059428 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059432 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059461 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059465 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059469 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059471 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059474 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059477 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059479 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059482 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059485 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.059917 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059487 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059490 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059493 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059496 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059498 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059501 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059503 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059506 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059508 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059511 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059513 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059516 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059518 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059520 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059523 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059526 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059530 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059534 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059537 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059541 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.060450 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059544 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059547 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059550 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059552 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059555 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059557 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059560 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059562 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059565 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059568 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059570 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059572 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059575 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059579 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059582 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059584 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059586 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059589 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059592 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059594 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.060951 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059597 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059599 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059602 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059604 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059607 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059610 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059613 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059616 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059634 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059638 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059641 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059645 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059647 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059650 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059653 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059655 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059658 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.059661 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060348 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.061453 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060355 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060360 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060364 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060367 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060371 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060374 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060377 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060380 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060383 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060386 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060389 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060392 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060395 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060398 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060401 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060404 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060407 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060410 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060413 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060416 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.061936 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060420 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060423 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060426 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060429 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060432 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060435 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060438 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060441 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060444 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060446 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060449 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060452 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060455 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060457 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060460 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060463 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060465 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060468 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060471 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.062435 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060473 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060476 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060479 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060481 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060484 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060486 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060489 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060491 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060495 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060499 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060502 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060505 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060510 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060513 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060516 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060520 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060523 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060525 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060529 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.062942 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060531 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060534 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060537 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060540 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060543 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060545 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060548 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060551 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060553 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060556 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060559 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060562 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060565 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060568 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060571 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060574 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060578 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060580 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060584 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060586 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.063403 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060589 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060592 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060594 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060597 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060601 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060604 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060607 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060610 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060612 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060615 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060631 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060634 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060636 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060639 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060641 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060644 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060647 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060649 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060652 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.060654 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.063908 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061038 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061043 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061046 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061049 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061054 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061058 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061061 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061064 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061066 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061069 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061080 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061083 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061085 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061088 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061091 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061093 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061098 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061101 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061104 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.064386 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061106 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061109 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061111 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061114 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061117 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061119 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061122 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061125 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061127 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061130 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061132 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061135 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061138 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061140 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061143 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061145 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061148 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061150 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061154 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061157 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.064855 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061160 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061162 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061165 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061167 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061169 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061172 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061175 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061177 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061179 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061183 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061186 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061189 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061191 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061194 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061196 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061199 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061201 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061204 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061206 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061209 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.065342 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061211 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061214 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061217 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061219 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061222 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061224 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061226 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061229 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061232 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061234 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061237 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061240 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061242 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061246 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061249 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061251 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061254 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061256 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061260 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061263 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.065853 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061267 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061271 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061275 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061278 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061281 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061283 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061286 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061289 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061292 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061296 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061298 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061301 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061303 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061306 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061308 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061311 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061314 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061316 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061318 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.061321 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061388 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:44:30.066361 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061432 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061439 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061443 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061448 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061452 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061457 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061461 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061464 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061467 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061471 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061474 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061477 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061481 2559 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061484 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061487 2559 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061490 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061493 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061497 2559 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061500 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061503 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061506 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061510 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061513 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:44:30.066884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061516 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061519 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061523 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061526 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061529 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061532 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061535 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061538 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061542 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061545 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061548 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061551 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061553 2559 flags.go:64] FLAG: --enable-server="true" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061558 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061562 2559 flags.go:64] FLAG: --event-burst="100" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061565 2559 flags.go:64] FLAG: --event-qps="50" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061568 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061571 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061574 2559 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061578 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061580 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061583 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061586 2559 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061589 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061592 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:44:30.067427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061594 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061597 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061600 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061603 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061606 2559 flags.go:64] FLAG: --feature-gates="" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061609 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061612 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061615 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061631 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061635 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061654 2559 flags.go:64] FLAG: --help="false" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061657 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061661 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061664 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061667 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061670 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061673 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061676 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061679 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061681 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061685 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061689 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061692 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061695 2559 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:44:30.068108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061698 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061701 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061704 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061707 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061710 2559 flags.go:64] FLAG: --lock-file="" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061713 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061715 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061718 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061724 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061726 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061729 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061732 2559 flags.go:64] FLAG: --logging-format="text" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061735 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061738 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061741 2559 flags.go:64] FLAG: --manifest-url="" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061744 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061748 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061751 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061755 2559 flags.go:64] FLAG: --max-pods="110" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061758 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061761 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061764 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061767 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061770 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061774 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:44:30.068704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061776 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061784 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061787 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061790 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061793 2559 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061796 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061802 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061805 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061808 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061811 2559 flags.go:64] FLAG: --port="10250" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061814 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061817 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0133758656127584f" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061820 2559 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061823 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061826 2559 flags.go:64] FLAG: --register-node="true" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061829 2559 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061835 2559 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061838 2559 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061841 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061843 2559 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061847 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061850 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061853 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061856 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061859 2559 flags.go:64] FLAG: --runonce="false" Sep 16 18:44:30.069520 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061862 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061865 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061869 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061872 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061875 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061879 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061882 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061886 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061888 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061891 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061894 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061897 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061900 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061903 2559 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061906 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061912 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061914 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061917 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061921 2559 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061924 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061927 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061929 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061932 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061935 2559 flags.go:64] FLAG: --v="2" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061939 2559 flags.go:64] FLAG: --version="false" Sep 16 18:44:30.070599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061943 2559 flags.go:64] FLAG: --vmodule="" Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061947 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.061950 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062068 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062072 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062075 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062078 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062080 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062083 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062086 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062088 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062091 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062095 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062098 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062101 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062103 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062106 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062109 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062113 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062115 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.071697 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062118 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062121 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062124 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062127 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062129 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062132 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062135 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062139 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062142 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062145 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062147 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062150 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062153 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062155 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062158 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062162 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062166 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062169 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062172 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.072559 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062175 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062178 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062181 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062183 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062186 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062190 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062193 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062196 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062198 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062201 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062203 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062207 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062211 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062215 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062218 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062221 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062224 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062227 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062230 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062232 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.073226 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062235 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062238 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062240 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062243 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062246 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062248 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062251 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062254 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062256 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062259 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062262 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062264 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062267 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062269 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062272 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062275 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062277 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062281 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062284 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062286 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.073891 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062289 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062292 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062295 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062299 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062301 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062304 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062307 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062309 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062313 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062316 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062318 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062321 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062323 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062326 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062329 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062331 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062334 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062337 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062339 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062342 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.074821 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062344 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062347 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.062350 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.063040 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.071444 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.071462 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.071487 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.071500 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071585 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071592 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071597 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071602 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071606 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071611 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071615 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.075461 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071638 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071643 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071647 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071652 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071656 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071660 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071664 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071668 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071673 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071677 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071681 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071686 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071690 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071695 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071699 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071702 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071706 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071710 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071714 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071718 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.075899 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071722 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071727 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071733 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071738 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071742 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071747 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071751 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071755 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071759 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071764 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071768 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071774 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071781 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071786 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071791 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071795 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071799 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071804 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071808 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.076610 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071812 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071816 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071820 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071824 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071829 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071833 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071838 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071842 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071846 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071850 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071854 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071858 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071863 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071868 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071872 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071876 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071882 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071886 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071892 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071898 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.077136 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071902 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071907 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071911 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071915 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071919 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071924 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071928 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071932 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071937 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071943 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071949 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071954 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071959 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071964 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071968 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071973 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071977 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071982 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071986 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071990 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.077660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071995 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.071999 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072004 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072008 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072012 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072017 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072021 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072026 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072030 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072035 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072039 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072043 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072048 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.072056 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072242 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:30.078191 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072252 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072257 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072263 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072267 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072272 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072279 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072285 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072290 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072294 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072298 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072302 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072307 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072311 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072315 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072319 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072323 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072328 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072332 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072336 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072341 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:30.078568 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072345 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072349 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072353 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072358 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072363 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072367 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072372 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072377 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072381 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072385 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072389 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072393 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072397 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072401 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072406 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072410 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072414 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072418 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072422 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072427 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:30.079134 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072431 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072435 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072440 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072444 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072450 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072456 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072460 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072465 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072470 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072475 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072479 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072484 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072488 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072493 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072498 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072502 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072507 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072512 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072516 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:30.079632 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072519 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072524 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072529 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072533 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072537 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072541 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072545 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072550 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072554 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072558 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072562 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072566 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072569 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072573 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072577 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072581 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072585 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072590 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072594 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072598 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:30.080101 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072602 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072608 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072615 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072634 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072639 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072643 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072647 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072652 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072657 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072661 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072665 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072670 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072674 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072678 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072684 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072689 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072693 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072697 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:30.080595 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:30.072702 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.072710 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.072857 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.076303 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.077654 2559 server.go:1019] "Starting client certificate rotation" Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.077754 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.077786 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:44:30.081049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.080396 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:44:30.081564 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.081547 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:44:30.094267 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.094244 2559 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:44:30.097475 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.097461 2559 log.go:25] "Validated CRI v1 image API" Sep 16 18:44:30.098337 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.098322 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:44:30.104079 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.104063 2559 fs.go:135] Filesystem UUIDs: map[4cff2b9d-b49f-4f2c-8d99-986211cb0fde:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 af5f7823-840d-4a93-8216-63d9ed03417c:/dev/nvme0n1p3] Sep 16 18:44:30.104132 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.104079 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:44:30.110470 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.110342 2559 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:30.107934334 +0000 UTC m=+0.372653997 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100495 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2905c84415fdd449f39e8fb9452b03 SystemUUID:ec2905c8-4415-fdd4-49f3-9e8fb9452b03 BootID:0cd20842-67cb-4c33-85ed-9cba9d8e3bee Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:a2:bd:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:a2:bd:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:49:47:3d:54:d5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:44:30.110470 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.110464 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:44:30.110581 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.110544 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:44:30.112128 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.112114 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:44:30.113282 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.113255 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:44:30.113421 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.113285 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:44:30.114189 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.114179 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:44:30.114220 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.114191 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:44:30.114220 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.114203 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:44:30.114220 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.114217 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:44:30.115631 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.115611 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:44:30.120928 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.120917 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:44:30.121036 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121027 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:44:30.121065 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121053 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:44:30.121065 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121062 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:44:30.121121 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121099 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:44:30.121121 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121110 2559 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:44:30.121214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121125 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:44:30.121531 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121506 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:44:30.121656 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.121538 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:44:30.125945 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.125919 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:44:30.128946 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.128927 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:44:30.129025 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.128975 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:44:30.131161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131147 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131169 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131178 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131186 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131198 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131217 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:44:30.131238 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131240 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:44:30.131414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131248 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:44:30.131414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131258 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:44:30.131414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131266 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:44:30.131414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131278 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:44:30.131414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.131287 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:44:30.132289 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.132275 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:44:30.132289 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.132288 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:44:30.132545 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.132526 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:44:30.132662 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.132635 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:44:30.136086 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.136072 2559 server.go:1268] "Started kubelet" Sep 16 18:44:30.136203 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.136149 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:44:30.136307 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.136225 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:44:30.136422 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.136166 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:44:30.136965 ip-10-0-129-152 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:44:30.137652 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.137633 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:44:30.138796 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.138779 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:44:30.140397 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.140374 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:44:30.142571 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.142553 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:44:30.143683 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.142570 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-152.ec2.internal.18d5e1697d15f25b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-152.ec2.internal,UID:ip-10-0-129-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-152.ec2.internal,},FirstTimestamp:2026-09-16 18:44:30.136046171 +0000 UTC m=+0.400765834,LastTimestamp:2026-09-16 18:44:30.136046171 +0000 UTC m=+0.400765834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-152.ec2.internal,}" Sep 16 18:44:30.144548 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.144532 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:44:30.144642 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.144547 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:44:30.144642 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.144614 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.145405 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145390 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:44:30.145508 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145399 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:44:30.145508 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145509 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:44:30.145817 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145799 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:44:30.145817 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145818 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:44:30.145921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145869 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:44:30.145921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145882 2559 factory.go:55] Registering systemd factory Sep 16 18:44:30.145921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.145890 2559 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:44:30.146123 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.146105 2559 factory.go:153] Registering CRI-O factory Sep 16 18:44:30.146123 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.146123 2559 factory.go:223] Registration of the crio container factory successfully Sep 16 18:44:30.146270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.146139 2559 factory.go:103] Registering Raw factory Sep 16 18:44:30.146270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.146149 2559 manager.go:1196] Started watching for new ooms in manager Sep 16 18:44:30.146780 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.146762 2559 manager.go:349] Starting recovery of all containers Sep 16 18:44:30.150471 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.150443 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:44:30.150752 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.150724 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:44:30.151970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.151939 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdbcs" Sep 16 18:44:30.158839 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.158823 2559 manager.go:354] Recovery completed Sep 16 18:44:30.165228 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.165215 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.165916 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.165898 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdbcs" Sep 16 18:44:30.167104 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167085 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.167165 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167115 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.167165 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167126 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.167718 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167702 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:44:30.167718 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167718 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:44:30.167859 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167724 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.167859 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.167735 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:44:30.168455 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.168442 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.168508 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.168466 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.168508 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.168490 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.169265 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.169203 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-152.ec2.internal.18d5e1697eefd566 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-152.ec2.internal,UID:ip-10-0-129-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-152.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-152.ec2.internal,},FirstTimestamp:2026-09-16 18:44:30.167102822 +0000 UTC m=+0.431822484,LastTimestamp:2026-09-16 18:44:30.167102822 +0000 UTC m=+0.431822484,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-152.ec2.internal,}" Sep 16 18:44:30.169873 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.169859 2559 policy_none.go:49] "None policy: Start" Sep 16 18:44:30.169921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.169880 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:44:30.169921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.169893 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:44:30.172154 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.172045 2559 policy_none.go:47] "Start" Sep 16 18:44:30.198372 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198357 2559 manager.go:337] "Starting Device Plugin manager" Sep 16 18:44:30.198448 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.198397 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:44:30.198448 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198410 2559 server.go:87] "Starting device plugin registration server" Sep 16 18:44:30.198704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198691 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:44:30.198778 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198707 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:44:30.199132 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198890 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:44:30.199132 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198969 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:44:30.199132 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.198978 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:44:30.204827 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.204808 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:44:30.227021 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.204861 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.246526 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.246502 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:44:30.247635 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.247607 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:44:30.247695 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.247672 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:44:30.247740 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.247697 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:44:30.247794 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.247769 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:44:30.250632 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.250595 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:30.299057 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.299036 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.299930 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.299915 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.300002 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.299945 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.300002 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.299959 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.300002 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.299983 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.308600 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.308548 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.308600 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.308568 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-152.ec2.internal\": node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.332119 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.332099 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.348501 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.348482 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal"] Sep 16 18:44:30.348562 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.348553 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.349328 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.349313 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.349388 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.349341 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.349388 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.349355 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.351759 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.351747 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.351880 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.351863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.351938 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.351910 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.352434 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352419 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.352491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352440 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.352491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352448 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.352491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352457 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.352491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352461 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.352491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.352466 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.354756 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.354738 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.354849 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.354763 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:30.355423 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.355407 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:30.355494 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.355437 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:30.355494 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.355451 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:30.385254 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.385235 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-152.ec2.internal\" not found" node="ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.388537 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.388522 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-152.ec2.internal\" not found" node="ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.432181 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.432164 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.446449 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.446432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b66b66cfebc8e98c7f82d9151ac864b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-152.ec2.internal\" (UID: \"5b66b66cfebc8e98c7f82d9151ac864b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.446511 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.446458 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.446511 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.446476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.532245 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.532215 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.547629 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.547688 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.547688 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b66b66cfebc8e98c7f82d9151ac864b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-152.ec2.internal\" (UID: \"5b66b66cfebc8e98c7f82d9151ac864b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.547750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.547750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92fd3c95dce3a13fa93c9500f2315fde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal\" (UID: \"92fd3c95dce3a13fa93c9500f2315fde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.547750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.547736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b66b66cfebc8e98c7f82d9151ac864b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-152.ec2.internal\" (UID: \"5b66b66cfebc8e98c7f82d9151ac864b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.632544 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.632487 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.687983 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.687963 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.691065 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:30.691050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:30.732907 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.732880 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:30.833474 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:30.833446 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.033665 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.033593 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.082015 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.081991 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:32.134630 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.134589 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.177166 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.177135 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:44:32.177691 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.177287 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:44:32.177691 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.177310 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:44:32.235585 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.235560 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.271880 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.271843 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:31 +0000 UTC" deadline="2028-03-07 18:21:53.412123575 +0000 UTC" Sep 16 18:44:32.272002 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.271880 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12911h37m21.14024806s" Sep 16 18:44:32.336159 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.336132 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.370162 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.370143 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:32.375128 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:32.375094 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fd3c95dce3a13fa93c9500f2315fde.slice/crio-cbf99c4e8c6e1e88804512e9bc5ec9fcc1cb4c820ec2fb414eae2fcd2446c97e WatchSource:0}: Error finding container cbf99c4e8c6e1e88804512e9bc5ec9fcc1cb4c820ec2fb414eae2fcd2446c97e: Status 404 returned error can't find the container with id cbf99c4e8c6e1e88804512e9bc5ec9fcc1cb4c820ec2fb414eae2fcd2446c97e Sep 16 18:44:32.375777 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:32.375756 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b66b66cfebc8e98c7f82d9151ac864b.slice/crio-49a9430c6d39bacd339ee5a2a879fea1ed353192c746843a8005897c8701ea60 WatchSource:0}: Error finding container 49a9430c6d39bacd339ee5a2a879fea1ed353192c746843a8005897c8701ea60: Status 404 returned error can't find the container with id 49a9430c6d39bacd339ee5a2a879fea1ed353192c746843a8005897c8701ea60 Sep 16 18:44:32.379503 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.379486 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:44:32.436792 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.436761 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.537294 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.537266 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.637833 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:32.637748 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-152.ec2.internal\" not found" Sep 16 18:44:32.713426 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.713402 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:32.744780 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.744750 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" Sep 16 18:44:32.753555 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.753533 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:44:32.754599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.754574 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" Sep 16 18:44:32.768503 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:32.768372 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:44:33.222220 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.222194 2559 apiserver.go:52] "Watching apiserver" Sep 16 18:44:33.230097 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.230070 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:44:33.233216 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.233184 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal","openshift-multus/multus-additional-cni-plugins-n2khs","openshift-multus/network-metrics-daemon-n7qvp","openshift-network-operator/iptables-alerter-bmcld","kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4","openshift-cluster-node-tuning-operator/tuned-9c4jv","openshift-dns/node-resolver-2qwhj","openshift-image-registry/node-ca-rk528","openshift-multus/multus-lk2j8","openshift-network-diagnostics/network-check-target-vpt6j","openshift-ovn-kubernetes/ovnkube-node-mhx6h","kube-system/global-pull-secret-syncer-v9fdf","kube-system/konnectivity-agent-prgkg"] Sep 16 18:44:33.235938 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.235914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.238071 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.238041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:33.238176 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.238129 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:33.238440 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.238422 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.238440 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.238434 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-46hks\"" Sep 16 18:44:33.238589 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.238577 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.240359 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.240338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.240453 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.240410 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:33.242424 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.242405 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.244457 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.244438 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:44:33.245116 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.245097 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.245196 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.245165 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.245358 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.245107 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:44:33.245474 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.245107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mr2vk\"" Sep 16 18:44:33.245640 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.245616 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.248046 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.248028 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6z7d\"" Sep 16 18:44:33.248229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.248213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.248301 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.248272 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:44:33.248536 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.248216 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:44:33.250189 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.250167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:44:33.250388 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.250370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.250778 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.250760 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x7tp8\"" Sep 16 18:44:33.250857 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.250780 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.250857 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.250812 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.252224 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.252203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.252308 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.252271 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:33.252462 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.252444 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:44:33.252529 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.252512 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.253227 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.252874 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.253227 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.253097 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vgkvj\"" Sep 16 18:44:33.255106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.255090 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.256061 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:33.256170 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61362957-46d6-4fbd-8806-6373375c63a5-host-slash\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.256170 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256122 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxms6\" (UniqueName: \"kubernetes.io/projected/61362957-46d6-4fbd-8806-6373375c63a5-kube-api-access-kxms6\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.256170 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256197 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d498543-f479-431b-b239-f609a6704a22-agent-certs\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256235 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256267 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-device-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysconfig\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256315 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-systemd\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.256413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256401 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52kj\" (UniqueName: \"kubernetes.io/projected/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-kube-api-access-b52kj\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.256734 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.256734 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.256734 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.256503 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61362957-46d6-4fbd-8806-6373375c63a5-iptables-alerter-script\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.257365 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d498543-f479-431b-b239-f609a6704a22-konnectivity-ca\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.257365 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-modprobe-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257365 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-run\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257365 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-lib-modules\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257365 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-host\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgkh\" (UniqueName: \"kubernetes.io/projected/35ce2f9c-e51f-46b5-af12-861b12925f3b-kube-api-access-9tgkh\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257380 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-conf\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdck\" (UniqueName: \"kubernetes.io/projected/aef78604-30a7-4e23-9ddf-9c726a7a5f30-kube-api-access-spdck\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-kubernetes\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-sys\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-tmp\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.257648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257654 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257799 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-var-lib-kubelet\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257820 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-tuned\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.257842 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.258094 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.258035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lx79v\"" Sep 16 18:44:33.258621 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.258600 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.261214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.261193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.262744 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.262724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rfnnz\"" Sep 16 18:44:33.262827 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.262805 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:44:33.263621 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.263601 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mvcs8\"" Sep 16 18:44:33.263705 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.263623 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:44:33.263705 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.263630 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.263881 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.263857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.264213 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.263867 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.266610 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.266593 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:44:33.267523 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267506 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:44:33.267523 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267516 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:44:33.267661 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267553 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:44:33.267720 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267700 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:44:33.267768 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267752 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r75fm\"" Sep 16 18:44:33.267814 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.267800 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:44:33.288141 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.288119 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2g9cz" Sep 16 18:44:33.298773 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.298755 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2g9cz" Sep 16 18:44:33.346195 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.346168 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:44:33.351577 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.351531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" event={"ID":"92fd3c95dce3a13fa93c9500f2315fde","Type":"ContainerStarted","Data":"cbf99c4e8c6e1e88804512e9bc5ec9fcc1cb4c820ec2fb414eae2fcd2446c97e"} Sep 16 18:44:33.352743 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.352719 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" event={"ID":"5b66b66cfebc8e98c7f82d9151ac864b","Type":"ContainerStarted","Data":"49a9430c6d39bacd339ee5a2a879fea1ed353192c746843a8005897c8701ea60"} Sep 16 18:44:33.358948 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.358927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-lib-modules\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.358958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-multus\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.359049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.358986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-daemon-config\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.359049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-multus-certs\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.359049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-log-socket\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-netd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-ovn\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-dbus\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-lib-modules\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.359229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-kubernetes\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-sys\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-tmp\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2lh\" (UniqueName: \"kubernetes.io/projected/75eea085-6c20-456f-b234-ef793183e012-kube-api-access-zc2lh\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-sys\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359414 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-tuned\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-cnibin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-systemd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14e13b94-9f75-43d9-a98b-8899a3926385-hosts-file\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61362957-46d6-4fbd-8806-6373375c63a5-host-slash\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.359594 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-system-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61362957-46d6-4fbd-8806-6373375c63a5-host-slash\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359629 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-netns\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359652 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359656 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-node-log\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359711 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5826602-57d8-4dc5-85e7-40ff7790798a-ovn-node-metrics-cert\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-device-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359794 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysconfig\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-socket-dir-parent\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.359850 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.359865 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-env-overrides\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.359934 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.859911677 +0000 UTC m=+3.024932012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:33.360214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-device-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysconfig\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360095 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-kubernetes\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-bin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-kubelet\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-hostroot\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35ce2f9c-e51f-46b5-af12-861b12925f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-systemd-units\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-var-lib-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360461 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61362957-46d6-4fbd-8806-6373375c63a5-iptables-alerter-script\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-modprobe-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-host\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.360973 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-cni-binary-copy\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.360586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361243 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-modprobe-d\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-host\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-config\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-script-lib\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361456 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbn7\" (UniqueName: \"kubernetes.io/projected/b5826602-57d8-4dc5-85e7-40ff7790798a-kube-api-access-zvbn7\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgkh\" (UniqueName: \"kubernetes.io/projected/35ce2f9c-e51f-46b5-af12-861b12925f3b-kube-api-access-9tgkh\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-conf\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361557 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdck\" (UniqueName: \"kubernetes.io/projected/aef78604-30a7-4e23-9ddf-9c726a7a5f30-kube-api-access-spdck\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-k8s-cni-cncf-io\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.361712 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61362957-46d6-4fbd-8806-6373375c63a5-iptables-alerter-script\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-sysctl-conf\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-kubelet\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.361979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c624e892-e030-4e19-a78c-c9a24b6abc1b-host\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbdp\" (UniqueName: \"kubernetes.io/projected/c624e892-e030-4e19-a78c-c9a24b6abc1b-kube-api-access-9pbdp\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-netns\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-slash\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-kubelet-config\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tlw\" (UniqueName: \"kubernetes.io/projected/14e13b94-9f75-43d9-a98b-8899a3926385-kube-api-access-96tlw\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsdv\" (UniqueName: \"kubernetes.io/projected/af372705-98be-4c8f-8cdb-c56f8735b57f-kube-api-access-pfsdv\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c624e892-e030-4e19-a78c-c9a24b6abc1b-serviceca\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.362280 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e13b94-9f75-43d9-a98b-8899a3926385-tmp-dir\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-var-lib-kubelet\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-bin\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362368 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-system-cni-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxms6\" (UniqueName: \"kubernetes.io/projected/61362957-46d6-4fbd-8806-6373375c63a5-kube-api-access-kxms6\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-cnibin\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d498543-f479-431b-b239-f609a6704a22-agent-certs\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-systemd\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-conf-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52kj\" (UniqueName: \"kubernetes.io/projected/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-kube-api-access-b52kj\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-etc-kubernetes\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-etc-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d498543-f479-431b-b239-f609a6704a22-konnectivity-ca\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.362821 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-run\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.363531 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.362898 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-run\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.363531 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.363081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-var-lib-kubelet\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.364409 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.364377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-systemd\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.364497 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.364416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-etc-tuned\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.366461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.364855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef78604-30a7-4e23-9ddf-9c726a7a5f30-tmp\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.366461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.365034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d498543-f479-431b-b239-f609a6704a22-konnectivity-ca\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.367530 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.367504 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d498543-f479-431b-b239-f609a6704a22-agent-certs\") pod \"konnectivity-agent-prgkg\" (UID: \"9d498543-f479-431b-b239-f609a6704a22\") " pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.369947 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.369928 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:33.370047 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.369952 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:33.370047 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.369966 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:33.370047 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.370030 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.870012156 +0000 UTC m=+3.035032477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:33.370902 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.370878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdck\" (UniqueName: \"kubernetes.io/projected/aef78604-30a7-4e23-9ddf-9c726a7a5f30-kube-api-access-spdck\") pod \"tuned-9c4jv\" (UID: \"aef78604-30a7-4e23-9ddf-9c726a7a5f30\") " pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.372169 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.372126 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxms6\" (UniqueName: \"kubernetes.io/projected/61362957-46d6-4fbd-8806-6373375c63a5-kube-api-access-kxms6\") pod \"iptables-alerter-bmcld\" (UID: \"61362957-46d6-4fbd-8806-6373375c63a5\") " pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.372511 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.372490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52kj\" (UniqueName: \"kubernetes.io/projected/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-kube-api-access-b52kj\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.374188 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.374163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgkh\" (UniqueName: \"kubernetes.io/projected/35ce2f9c-e51f-46b5-af12-861b12925f3b-kube-api-access-9tgkh\") pod \"aws-ebs-csi-driver-node-c2kb4\" (UID: \"35ce2f9c-e51f-46b5-af12-861b12925f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.463034 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.463034 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tlw\" (UniqueName: \"kubernetes.io/projected/14e13b94-9f75-43d9-a98b-8899a3926385-kube-api-access-96tlw\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsdv\" (UniqueName: \"kubernetes.io/projected/af372705-98be-4c8f-8cdb-c56f8735b57f-kube-api-access-pfsdv\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c624e892-e030-4e19-a78c-c9a24b6abc1b-serviceca\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e13b94-9f75-43d9-a98b-8899a3926385-tmp-dir\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-bin\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-system-cni-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.463165 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-cnibin\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463204 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-conf-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463232 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.463232 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.963213331 +0000 UTC m=+3.128233654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463254 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-conf-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-etc-kubernetes\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-etc-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-multus\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-daemon-config\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-multus-certs\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-etc-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-log-socket\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-netd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-log-socket\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-ovn\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-dbus\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.463611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2lh\" (UniqueName: \"kubernetes.io/projected/75eea085-6c20-456f-b234-ef793183e012-kube-api-access-zc2lh\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-cnibin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-systemd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14e13b94-9f75-43d9-a98b-8899a3926385-hosts-file\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-system-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-netns\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-node-log\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5826602-57d8-4dc5-85e7-40ff7790798a-ovn-node-metrics-cert\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-socket-dir-parent\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-env-overrides\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c624e892-e030-4e19-a78c-c9a24b6abc1b-serviceca\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-bin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.463992 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-kubelet\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-hostroot\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464136 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-systemd-units\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-var-lib-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-var-lib-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-cni-binary-copy\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-config\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-netd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e13b94-9f75-43d9-a98b-8899a3926385-tmp-dir\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-script-lib\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbn7\" (UniqueName: \"kubernetes.io/projected/b5826602-57d8-4dc5-85e7-40ff7790798a-kube-api-access-zvbn7\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464252 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-ovn\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-cnibin\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-node-log\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.464892 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-k8s-cni-cncf-io\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-kubelet\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c624e892-e030-4e19-a78c-c9a24b6abc1b-host\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbdp\" (UniqueName: \"kubernetes.io/projected/c624e892-e030-4e19-a78c-c9a24b6abc1b-kube-api-access-9pbdp\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-netns\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-slash\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-kubelet-config\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-kubelet-config\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-multus\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-systemd\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-cni-bin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14e13b94-9f75-43d9-a98b-8899a3926385-hosts-file\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-system-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-cnibin\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-run-netns\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-hostroot\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-var-lib-kubelet\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.465722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-etc-kubernetes\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-socket-dir-parent\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-daemon-config\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-multus-certs\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-env-overrides\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465250 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-cni-bin\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-system-cni-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75eea085-6c20-456f-b234-ef793183e012-cni-binary-copy\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465305 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-multus-cni-dir\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-slash\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-run-openvswitch\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-systemd-units\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.464475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75eea085-6c20-456f-b234-ef793183e012-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-netns\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84b69c03-718b-41c6-be39-cbc758435e44-dbus\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465452 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af372705-98be-4c8f-8cdb-c56f8735b57f-host-run-k8s-cni-cncf-io\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5826602-57d8-4dc5-85e7-40ff7790798a-host-kubelet\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.466570 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c624e892-e030-4e19-a78c-c9a24b6abc1b-host\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.467461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-config\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.467461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5826602-57d8-4dc5-85e7-40ff7790798a-ovnkube-script-lib\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.467461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.465836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af372705-98be-4c8f-8cdb-c56f8735b57f-cni-binary-copy\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.467461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.466966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5826602-57d8-4dc5-85e7-40ff7790798a-ovn-node-metrics-cert\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.471910 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.471876 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsdv\" (UniqueName: \"kubernetes.io/projected/af372705-98be-4c8f-8cdb-c56f8735b57f-kube-api-access-pfsdv\") pod \"multus-lk2j8\" (UID: \"af372705-98be-4c8f-8cdb-c56f8735b57f\") " pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.472041 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.471912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tlw\" (UniqueName: \"kubernetes.io/projected/14e13b94-9f75-43d9-a98b-8899a3926385-kube-api-access-96tlw\") pod \"node-resolver-2qwhj\" (UID: \"14e13b94-9f75-43d9-a98b-8899a3926385\") " pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.472630 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.472583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2lh\" (UniqueName: \"kubernetes.io/projected/75eea085-6c20-456f-b234-ef793183e012-kube-api-access-zc2lh\") pod \"multus-additional-cni-plugins-n2khs\" (UID: \"75eea085-6c20-456f-b234-ef793183e012\") " pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.474169 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.474149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbn7\" (UniqueName: \"kubernetes.io/projected/b5826602-57d8-4dc5-85e7-40ff7790798a-kube-api-access-zvbn7\") pod \"ovnkube-node-mhx6h\" (UID: \"b5826602-57d8-4dc5-85e7-40ff7790798a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.474521 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.474502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbdp\" (UniqueName: \"kubernetes.io/projected/c624e892-e030-4e19-a78c-c9a24b6abc1b-kube-api-access-9pbdp\") pod \"node-ca-rk528\" (UID: \"c624e892-e030-4e19-a78c-c9a24b6abc1b\") " pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.548563 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.548530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" Sep 16 18:44:33.556405 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.556382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bmcld" Sep 16 18:44:33.567397 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.567377 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:33.571114 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.571093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" Sep 16 18:44:33.577741 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.577721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk528" Sep 16 18:44:33.586412 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.586395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n2khs" Sep 16 18:44:33.593373 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.593352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lk2j8" Sep 16 18:44:33.602116 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.602099 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2qwhj" Sep 16 18:44:33.606841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.606822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:33.765709 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.765633 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:33.867120 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.867093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:33.867273 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.867204 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:33.867273 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.867268 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.867250497 +0000 UTC m=+4.032270835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:33.967885 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.967855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:33.967885 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:33.967891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968001 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968014 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968030 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968043 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968069 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.96805099 +0000 UTC m=+4.133071323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:33.968132 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:33.968088 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:34.968075174 +0000 UTC m=+4.133095494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:34.227073 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.227045 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e13b94_9f75_43d9_a98b_8899a3926385.slice/crio-4da29fc50ee44762f574cb84ef244dc8bee57b0851fd3b014856366bdc33bf93 WatchSource:0}: Error finding container 4da29fc50ee44762f574cb84ef244dc8bee57b0851fd3b014856366bdc33bf93: Status 404 returned error can't find the container with id 4da29fc50ee44762f574cb84ef244dc8bee57b0851fd3b014856366bdc33bf93 Sep 16 18:44:34.227660 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.227638 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ce2f9c_e51f_46b5_af12_861b12925f3b.slice/crio-433529afca74e5e211037ec45bd5a27480d3e222800fb801adbb0c7d35bff31f WatchSource:0}: Error finding container 433529afca74e5e211037ec45bd5a27480d3e222800fb801adbb0c7d35bff31f: Status 404 returned error can't find the container with id 433529afca74e5e211037ec45bd5a27480d3e222800fb801adbb0c7d35bff31f Sep 16 18:44:34.229948 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.229894 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc624e892_e030_4e19_a78c_c9a24b6abc1b.slice/crio-7fffa648912a59d7cb232614257de8d83d9f2f1971889881904f856fed54f28e WatchSource:0}: Error finding container 7fffa648912a59d7cb232614257de8d83d9f2f1971889881904f856fed54f28e: Status 404 returned error can't find the container with id 7fffa648912a59d7cb232614257de8d83d9f2f1971889881904f856fed54f28e Sep 16 18:44:34.232246 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.232200 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eea085_6c20_456f_b234_ef793183e012.slice/crio-571ce972c90a265ae65efc467ad133396345c5308c27013d821bea3c1629d0a0 WatchSource:0}: Error finding container 571ce972c90a265ae65efc467ad133396345c5308c27013d821bea3c1629d0a0: Status 404 returned error can't find the container with id 571ce972c90a265ae65efc467ad133396345c5308c27013d821bea3c1629d0a0 Sep 16 18:44:34.232961 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.232877 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef78604_30a7_4e23_9ddf_9c726a7a5f30.slice/crio-95e8f3353eb920fa37e656408cefc6295521f807af3a748977f06fa39a67f5b3 WatchSource:0}: Error finding container 95e8f3353eb920fa37e656408cefc6295521f807af3a748977f06fa39a67f5b3: Status 404 returned error can't find the container with id 95e8f3353eb920fa37e656408cefc6295521f807af3a748977f06fa39a67f5b3 Sep 16 18:44:34.234406 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.234289 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf372705_98be_4c8f_8cdb_c56f8735b57f.slice/crio-a709156ceb2ba95f0a85d385b723c2ee33f271e171e471bcac85e78adaf23e79 WatchSource:0}: Error finding container a709156ceb2ba95f0a85d385b723c2ee33f271e171e471bcac85e78adaf23e79: Status 404 returned error can't find the container with id a709156ceb2ba95f0a85d385b723c2ee33f271e171e471bcac85e78adaf23e79 Sep 16 18:44:34.236416 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.236393 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5826602_57d8_4dc5_85e7_40ff7790798a.slice/crio-ce912ea37e3c6c3d449423ee80f16ee278e4c8a59393bdcb3d34a4f005420113 WatchSource:0}: Error finding container ce912ea37e3c6c3d449423ee80f16ee278e4c8a59393bdcb3d34a4f005420113: Status 404 returned error can't find the container with id ce912ea37e3c6c3d449423ee80f16ee278e4c8a59393bdcb3d34a4f005420113 Sep 16 18:44:34.236725 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:44:34.236618 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d498543_f479_431b_b239_f609a6704a22.slice/crio-e8a2f951691470cd29bd7e2ce119d6fe9adde90a65851b7a2b19ba8cade8594b WatchSource:0}: Error finding container e8a2f951691470cd29bd7e2ce119d6fe9adde90a65851b7a2b19ba8cade8594b: Status 404 returned error can't find the container with id e8a2f951691470cd29bd7e2ce119d6fe9adde90a65851b7a2b19ba8cade8594b Sep 16 18:44:34.302336 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.302123 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:33 +0000 UTC" deadline="2028-06-11 02:22:14.753514754 +0000 UTC" Sep 16 18:44:34.302434 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.302344 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15199h37m40.45117733s" Sep 16 18:44:34.357735 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.357692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lk2j8" event={"ID":"af372705-98be-4c8f-8cdb-c56f8735b57f","Type":"ContainerStarted","Data":"a709156ceb2ba95f0a85d385b723c2ee33f271e171e471bcac85e78adaf23e79"} Sep 16 18:44:34.358865 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.358839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" event={"ID":"aef78604-30a7-4e23-9ddf-9c726a7a5f30","Type":"ContainerStarted","Data":"95e8f3353eb920fa37e656408cefc6295521f807af3a748977f06fa39a67f5b3"} Sep 16 18:44:34.360796 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.360763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk528" event={"ID":"c624e892-e030-4e19-a78c-c9a24b6abc1b","Type":"ContainerStarted","Data":"7fffa648912a59d7cb232614257de8d83d9f2f1971889881904f856fed54f28e"} Sep 16 18:44:34.362497 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.362475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qwhj" event={"ID":"14e13b94-9f75-43d9-a98b-8899a3926385","Type":"ContainerStarted","Data":"4da29fc50ee44762f574cb84ef244dc8bee57b0851fd3b014856366bdc33bf93"} Sep 16 18:44:34.365636 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.365534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" event={"ID":"5b66b66cfebc8e98c7f82d9151ac864b","Type":"ContainerStarted","Data":"99ca7d925ba7fd721699143aca5f2ea4bafaab69393e9a863ecde1d61a50bd77"} Sep 16 18:44:34.369446 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.369422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bmcld" event={"ID":"61362957-46d6-4fbd-8806-6373375c63a5","Type":"ContainerStarted","Data":"bdcf795c42c37119c58fb4244e794b7d37118812c13e0a1b42b338bcd43f65f3"} Sep 16 18:44:34.371893 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.371862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prgkg" event={"ID":"9d498543-f479-431b-b239-f609a6704a22","Type":"ContainerStarted","Data":"e8a2f951691470cd29bd7e2ce119d6fe9adde90a65851b7a2b19ba8cade8594b"} Sep 16 18:44:34.373210 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.373184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"ce912ea37e3c6c3d449423ee80f16ee278e4c8a59393bdcb3d34a4f005420113"} Sep 16 18:44:34.374700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.374677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerStarted","Data":"571ce972c90a265ae65efc467ad133396345c5308c27013d821bea3c1629d0a0"} Sep 16 18:44:34.376683 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.376663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" event={"ID":"35ce2f9c-e51f-46b5-af12-861b12925f3b","Type":"ContainerStarted","Data":"433529afca74e5e211037ec45bd5a27480d3e222800fb801adbb0c7d35bff31f"} Sep 16 18:44:34.378644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.378594 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-152.ec2.internal" podStartSLOduration=2.3785766649999998 podStartE2EDuration="2.378576665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:34.37810612 +0000 UTC m=+3.543126462" watchObservedRunningTime="2026-09-16 18:44:34.378576665 +0000 UTC m=+3.543597009" Sep 16 18:44:34.874827 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.874790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:34.875018 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.874970 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:34.875076 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.875043 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.875015071 +0000 UTC m=+6.040035392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:34.904834 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.904782 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:34.976174 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.976124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:34.976370 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:34.976233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:34.976370 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976361 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:34.976488 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976421 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.976404809 +0000 UTC m=+6.141425129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:34.976836 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976818 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:34.976920 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976841 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:34.976920 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976852 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:34.976920 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:34.976897 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.97688208 +0000 UTC m=+6.141902423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:35.302766 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.302716 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:33 +0000 UTC" deadline="2028-06-25 06:32:42.035522678 +0000 UTC" Sep 16 18:44:35.302766 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.302759 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15539h48m6.732770243s" Sep 16 18:44:35.348484 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.348458 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:35.348651 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:35.348594 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:35.349022 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.349004 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:35.349201 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:35.349182 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:35.350834 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.350813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:35.350947 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:35.350904 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:35.386843 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.386811 2559 generic.go:356] "Generic (PLEG): container finished" podID="92fd3c95dce3a13fa93c9500f2315fde" containerID="257c616b31b085d0712898e00de5d4e71987005935d426265150fc521dd2eb48" exitCode=0 Sep 16 18:44:35.387005 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:35.386887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" event={"ID":"92fd3c95dce3a13fa93c9500f2315fde","Type":"ContainerDied","Data":"257c616b31b085d0712898e00de5d4e71987005935d426265150fc521dd2eb48"} Sep 16 18:44:36.403567 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:36.403485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" event={"ID":"92fd3c95dce3a13fa93c9500f2315fde","Type":"ContainerStarted","Data":"1a25fd42a7c68b0054b36ff999ff4d761d40d3a1b7a7a0d43d0f53a2547f01ce"} Sep 16 18:44:36.892787 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:36.892731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:36.893024 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.892968 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:36.893103 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.893036 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.893016489 +0000 UTC m=+10.058036829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:36.996549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:36.996601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.996763 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.996791 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.996803 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.996859 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.996841411 +0000 UTC m=+10.161861738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.997228 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:36.997304 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:36.997272 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.997258748 +0000 UTC m=+10.162279078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:37.348441 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:37.348411 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:37.348615 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:37.348530 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:37.348799 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:37.348775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:37.348906 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:37.348881 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:37.348990 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:37.348972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:37.349124 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:37.349087 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:39.347874 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:39.348005 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:39.348404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:39.348505 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:39.348551 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:39.348654 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:39.348619 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:40.932014 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:40.931974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:40.932597 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:40.932197 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:40.932597 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:40.932266 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:48.932248709 +0000 UTC m=+18.097269034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:41.033527 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:41.033466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:41.033527 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:41.033515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033617 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033659 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033676 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033688 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033697 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:49.03367662 +0000 UTC m=+18.198696957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:41.033828 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.033734 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:49.03372044 +0000 UTC m=+18.198740784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:41.348907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:41.349013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.349031 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.349121 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:41.349185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:41.349958 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:41.349266 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:43.348246 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:43.348210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:43.348716 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:43.348210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:43.348716 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:43.348308 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:43.348716 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:43.348210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:43.348716 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:43.348411 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:43.348716 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:43.348463 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:45.348190 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:45.348155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:45.348663 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:45.348213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:45.348663 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:45.348155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:45.348663 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:45.348289 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:45.348663 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:45.348380 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:45.348663 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:45.348456 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:47.348439 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:47.348403 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:47.348890 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:47.348408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:47.348890 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:47.348550 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:47.348890 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:47.348415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:47.348890 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:47.348621 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:47.348890 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:47.348697 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:48.990451 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:48.990413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:48.990925 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:48.990598 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:48.990925 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:48.990691 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.990668988 +0000 UTC m=+34.155689330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:49.091739 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:49.091697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:49.091899 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:49.091780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:49.091899 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091823 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:49.091899 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091848 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:49.091899 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091859 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:49.091899 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091867 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:49.092152 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091915 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.09190103 +0000 UTC m=+34.256921356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:49.092152 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.091932 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.091925541 +0000 UTC m=+34.256945861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:49.348286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:49.348252 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:49.348465 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.348376 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:49.348465 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:49.348252 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:49.348586 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.348510 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:49.348586 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:49.348548 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:49.348694 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:49.348602 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:51.348747 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:51.348714 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:51.349184 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:51.348791 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:51.349184 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:51.348870 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:51.349184 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:51.348903 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:51.349184 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:51.348961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:51.349184 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:51.349059 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:52.437032 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.436675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"61de3f0c26eb15c57d10fbf15cea29c45b6e8da8602f7a3536e371779c5e692b"} Sep 16 18:44:52.437583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.437120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"3f0c78b666649631f9011a68f8762127a693ca661a6e4eaae85c06afc130e62e"} Sep 16 18:44:52.437583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.437175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"6823f52de7c13112c95fa9ca1942a3c67398c22832bc0cdad034b28d5efe02bd"} Sep 16 18:44:52.437583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.437212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"482142794d7dbc9c98cd951bd518880911ec740833d2dd503cca90a151739ffe"} Sep 16 18:44:52.437583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.437254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"747a7201bcc57a1468f346821e52dc14441a44f22ef38c0122471ba7a4c07ef9"} Sep 16 18:44:52.437583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.437302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"29dd281fba391a27b9861a5974a720ead894545f940a102618016652f33f66b7"} Sep 16 18:44:52.438202 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.438182 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="952fc65917d53b2c791e804fa7b7dc49c18c13c938597bfb383ad79ca50b8351" exitCode=0 Sep 16 18:44:52.438277 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.438252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"952fc65917d53b2c791e804fa7b7dc49c18c13c938597bfb383ad79ca50b8351"} Sep 16 18:44:52.439812 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.439787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" event={"ID":"35ce2f9c-e51f-46b5-af12-861b12925f3b","Type":"ContainerStarted","Data":"82f06c9e948b279a6335327abc1a54aa6f7e280fc981b614391e83425c83bcca"} Sep 16 18:44:52.441230 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.441193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lk2j8" event={"ID":"af372705-98be-4c8f-8cdb-c56f8735b57f","Type":"ContainerStarted","Data":"54e60be306da2f3f9e950d23319bfd8c05c1cea03937746c2f4421dac96e753a"} Sep 16 18:44:52.442540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.442513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" event={"ID":"aef78604-30a7-4e23-9ddf-9c726a7a5f30","Type":"ContainerStarted","Data":"5293a46ae2e73babf4fb139f4d5ff24b1bb5afdf93fe0b1207abbec730c1846d"} Sep 16 18:44:52.443898 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.443879 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk528" event={"ID":"c624e892-e030-4e19-a78c-c9a24b6abc1b","Type":"ContainerStarted","Data":"178434c385e537ce8eb6f7cb24bbb5f09f57f19a4070cdbf16ea03cf27be2d69"} Sep 16 18:44:52.445299 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.445277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2qwhj" event={"ID":"14e13b94-9f75-43d9-a98b-8899a3926385","Type":"ContainerStarted","Data":"3222a7539f0a2ab0e28f18759770b75798f3fb3d3d9b1ace30de47e36f4e5009"} Sep 16 18:44:52.446639 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.446619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prgkg" event={"ID":"9d498543-f479-431b-b239-f609a6704a22","Type":"ContainerStarted","Data":"04abc6f6fd4e45662c2fea30a333b4ff45b6afac2802cbca25e553d38469ecfb"} Sep 16 18:44:52.460444 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.460403 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-152.ec2.internal" podStartSLOduration=20.460391316 podStartE2EDuration="20.460391316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:36.432480071 +0000 UTC m=+5.597500414" watchObservedRunningTime="2026-09-16 18:44:52.460391316 +0000 UTC m=+21.625411658" Sep 16 18:44:52.475233 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.475163 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rk528" podStartSLOduration=3.986382045 podStartE2EDuration="21.475144692s" totalImagesPullingTime="17.488762647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.474808223 +0000 UTC m=+21.639828599" watchObservedRunningTime="2026-09-16 18:44:52.475144692 +0000 UTC m=+21.640165034" Sep 16 18:44:52.493483 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.493448 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9c4jv" podStartSLOduration=4.010213997 podStartE2EDuration="21.493434477s" totalImagesPullingTime="17.48322048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.492873898 +0000 UTC m=+21.657894250" watchObservedRunningTime="2026-09-16 18:44:52.493434477 +0000 UTC m=+21.658454818" Sep 16 18:44:52.509201 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.509160 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-prgkg" podStartSLOduration=4.02979309 podStartE2EDuration="21.509144434s" totalImagesPullingTime="17.479351344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.508711459 +0000 UTC m=+21.673731802" watchObservedRunningTime="2026-09-16 18:44:52.509144434 +0000 UTC m=+21.674164775" Sep 16 18:44:52.529699 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.529648 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lk2j8" podStartSLOduration=3.989124131 podStartE2EDuration="21.529630938s" totalImagesPullingTime="17.540506807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.528951932 +0000 UTC m=+21.693972275" watchObservedRunningTime="2026-09-16 18:44:52.529630938 +0000 UTC m=+21.694651273" Sep 16 18:44:52.545443 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:52.545409 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2qwhj" podStartSLOduration=4.053378713 podStartE2EDuration="21.545397216s" totalImagesPullingTime="17.492018503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.544574001 +0000 UTC m=+21.709594348" watchObservedRunningTime="2026-09-16 18:44:52.545397216 +0000 UTC m=+21.710417558" Sep 16 18:44:53.180095 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.180075 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:44:53.310231 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.310076 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:53.180090563Z","UUID":"179f38fb-0fd5-495d-9e71-bb48bc6e9f32","Handler":null,"Name":"","Endpoint":""} Sep 16 18:44:53.311631 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.311575 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:44:53.311631 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.311608 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:44:53.348214 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.348180 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:53.348360 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.348190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:53.348360 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.348300 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:53.348360 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:53.348305 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:53.348483 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:53.348400 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:53.348483 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:53.348476 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:53.450376 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.450333 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bmcld" event={"ID":"61362957-46d6-4fbd-8806-6373375c63a5","Type":"ContainerStarted","Data":"ea8cc56d9732fa8dcbf7daae69ff61d3d9cecbc331fc2d0587f5a6f209dc7c88"} Sep 16 18:44:53.452663 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:53.452635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" event={"ID":"35ce2f9c-e51f-46b5-af12-861b12925f3b","Type":"ContainerStarted","Data":"e6445162cafdd11e163ea22cd5dccd04de3d1e7f64d19d51945213c92354e175"} Sep 16 18:44:54.802804 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:54.802557 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:54.803298 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:54.803216 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:54.820397 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:54.820307 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bmcld" podStartSLOduration=6.336689335 podStartE2EDuration="23.820292396s" totalImagesPullingTime="17.483603061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:53.473460753 +0000 UTC m=+22.638481095" watchObservedRunningTime="2026-09-16 18:44:54.820292396 +0000 UTC m=+23.985312737" Sep 16 18:44:55.348330 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.348292 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:55.348578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.348295 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:55.348578 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:55.348430 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:55.348578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.348303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:55.349386 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:55.349212 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:55.352573 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:55.349624 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:55.461870 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.461832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"48c39fb90ae93f3eb754c410d498600a9465faf1117119139c48c7eee2e37048"} Sep 16 18:44:55.464301 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.464235 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" event={"ID":"35ce2f9c-e51f-46b5-af12-861b12925f3b","Type":"ContainerStarted","Data":"f757d929828ac3c36f7d0ddb46dc4c671b900e2078729da8970743ca7514daeb"} Sep 16 18:44:55.464603 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.464569 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:55.465088 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.465067 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-prgkg" Sep 16 18:44:55.484721 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:55.484675 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2kb4" podStartSLOduration=4.574548654 podStartE2EDuration="24.484659955s" totalImagesPullingTime="19.910111301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.48402037 +0000 UTC m=+24.649040727" watchObservedRunningTime="2026-09-16 18:44:55.484659955 +0000 UTC m=+24.649680323" Sep 16 18:44:57.347760 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.347580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:57.348077 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.347580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:57.348077 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:57.347852 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:57.348077 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.347637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:57.348077 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:57.347929 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:57.348077 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:57.348045 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:57.472123 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.472084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" event={"ID":"b5826602-57d8-4dc5-85e7-40ff7790798a","Type":"ContainerStarted","Data":"77e212a817619dbea882a6028d5b928b48ced11320837e51bd49bce2e6f8c4e0"} Sep 16 18:44:57.472314 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.472296 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:57.473778 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.473752 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="c2551655503d41fc97cbe7590518c92930e5d50483d4ea8c72c61bc3643cf92c" exitCode=0 Sep 16 18:44:57.473883 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.473836 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"c2551655503d41fc97cbe7590518c92930e5d50483d4ea8c72c61bc3643cf92c"} Sep 16 18:44:57.487545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.487527 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:57.507708 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:57.507666 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" podStartSLOduration=8.965401497 podStartE2EDuration="26.507653958s" totalImagesPullingTime="17.542252461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:57.507212747 +0000 UTC m=+26.672233089" watchObservedRunningTime="2026-09-16 18:44:57.507653958 +0000 UTC m=+26.672674299" Sep 16 18:44:58.476884 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.476854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:58.477541 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.476968 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:58.492600 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.492578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:44:58.735375 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.735289 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9fdf"] Sep 16 18:44:58.735509 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.735437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:44:58.735639 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:58.735550 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:44:58.740299 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.740277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7qvp"] Sep 16 18:44:58.740466 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.740408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:44:58.740574 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:58.740510 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:44:58.770607 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.770573 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vpt6j"] Sep 16 18:44:58.770731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:58.770660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:44:58.770731 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:44:58.770724 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:44:59.480242 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:59.480208 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="b2425c293164b09bca5c9107e9b74f464acac5468d8800fe4da359593ce09141" exitCode=0 Sep 16 18:44:59.480849 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:44:59.480298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"b2425c293164b09bca5c9107e9b74f464acac5468d8800fe4da359593ce09141"} Sep 16 18:45:00.348628 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:00.348457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:00.348813 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:00.348476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:00.348813 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:00.348725 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:45:00.348813 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:00.348517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:00.348813 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:00.348783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:45:00.349033 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:00.348838 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:45:01.486546 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:01.486516 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="f22afaaf05ccffa0e37a900a10450274ac2fae503511537a0e8e70f3ea306f69" exitCode=0 Sep 16 18:45:01.486932 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:01.486567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"f22afaaf05ccffa0e37a900a10450274ac2fae503511537a0e8e70f3ea306f69"} Sep 16 18:45:02.348144 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:02.348115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:02.348355 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:02.348216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:02.348355 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:02.348224 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:45:02.348355 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:02.348245 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:02.348355 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:02.348341 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:45:02.348578 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:02.348394 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:45:04.348505 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.348467 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:04.348505 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.348498 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:04.348977 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.348500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:04.348977 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:04.348603 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vpt6j" podUID="3b9dd24b-6c17-4242-9237-c9a757140681" Sep 16 18:45:04.348977 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:04.348665 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7qvp" podUID="0c6db619-fa2b-4eae-9e3e-1b94fc47bba3" Sep 16 18:45:04.348977 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:04.348768 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9fdf" podUID="84b69c03-718b-41c6-be39-cbc758435e44" Sep 16 18:45:04.673917 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.673882 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-152.ec2.internal" event="NodeReady" Sep 16 18:45:04.674108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.674052 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:45:04.725485 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.725432 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w22ll"] Sep 16 18:45:04.728686 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.728661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.729718 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.729684 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k9wjj"] Sep 16 18:45:04.733851 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.733819 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:45:04.733975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.733862 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.733975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.733913 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g84ch\"" Sep 16 18:45:04.733975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.733956 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:45:04.736703 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.736664 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:45:04.737074 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.736996 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:45:04.737216 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.736996 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:45:04.737216 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.737121 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9lmtn\"" Sep 16 18:45:04.743170 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.743132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w22ll"] Sep 16 18:45:04.744037 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.744017 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k9wjj"] Sep 16 18:45:04.809150 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm94x\" (UniqueName: \"kubernetes.io/projected/772f55b7-eecf-4c9a-93a6-547a73a0b147-kube-api-access-mm94x\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.809303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772f55b7-eecf-4c9a-93a6-547a73a0b147-tmp-dir\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.809303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.809303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/772f55b7-eecf-4c9a-93a6-547a73a0b147-config-volume\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.809303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft76q\" (UniqueName: \"kubernetes.io/projected/a470b680-0106-40e5-aab1-71f7f33a0598-kube-api-access-ft76q\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.809535 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.809359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a470b680-0106-40e5-aab1-71f7f33a0598-cert\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.909933 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.909898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm94x\" (UniqueName: \"kubernetes.io/projected/772f55b7-eecf-4c9a-93a6-547a73a0b147-kube-api-access-mm94x\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.909933 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.909939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772f55b7-eecf-4c9a-93a6-547a73a0b147-tmp-dir\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.910163 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.910222 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/772f55b7-eecf-4c9a-93a6-547a73a0b147-config-volume\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.910222 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft76q\" (UniqueName: \"kubernetes.io/projected/a470b680-0106-40e5-aab1-71f7f33a0598-kube-api-access-ft76q\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.910341 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a470b680-0106-40e5-aab1-71f7f33a0598-cert\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.910341 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772f55b7-eecf-4c9a-93a6-547a73a0b147-tmp-dir\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.910341 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:04.910293 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:04.910485 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:04.910392 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.410368788 +0000 UTC m=+34.575389125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:04.910804 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.910782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/772f55b7-eecf-4c9a-93a6-547a73a0b147-config-volume\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:04.914578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.914555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a470b680-0106-40e5-aab1-71f7f33a0598-cert\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.920679 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.920656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft76q\" (UniqueName: \"kubernetes.io/projected/a470b680-0106-40e5-aab1-71f7f33a0598-kube-api-access-ft76q\") pod \"ingress-canary-k9wjj\" (UID: \"a470b680-0106-40e5-aab1-71f7f33a0598\") " pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:04.920783 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:04.920658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm94x\" (UniqueName: \"kubernetes.io/projected/772f55b7-eecf-4c9a-93a6-547a73a0b147-kube-api-access-mm94x\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:05.010928 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:05.010829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:05.011075 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.011005 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:05.011144 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.011092 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.011070938 +0000 UTC m=+66.176091274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:05.048620 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:05.048598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k9wjj" Sep 16 18:45:05.111975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:05.111941 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:05.112103 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:05.112052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:05.112165 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112138 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:05.112165 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112155 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:05.112165 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112164 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:05.112311 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112179 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9zst2 for pod openshift-network-diagnostics/network-check-target-vpt6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:05.112311 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112225 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret podName:84b69c03-718b-41c6-be39-cbc758435e44 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.11220844 +0000 UTC m=+66.277228775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret") pod "global-pull-secret-syncer-v9fdf" (UID: "84b69c03-718b-41c6-be39-cbc758435e44") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:05.112311 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.112245 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2 podName:3b9dd24b-6c17-4242-9237-c9a757140681 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.112235877 +0000 UTC m=+66.277256199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zst2" (UniqueName: "kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2") pod "network-check-target-vpt6j" (UID: "3b9dd24b-6c17-4242-9237-c9a757140681") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:05.414627 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:05.414588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:05.415079 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.414735 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:05.415079 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:05.414806 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:06.414785718 +0000 UTC m=+35.579806046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:06.093301 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.093099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k9wjj"] Sep 16 18:45:06.099661 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:45:06.099633 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda470b680_0106_40e5_aab1_71f7f33a0598.slice/crio-42d0e47cad5ae3698247b91810078152673888ecdce7bb867e9375b1e77ede57 WatchSource:0}: Error finding container 42d0e47cad5ae3698247b91810078152673888ecdce7bb867e9375b1e77ede57: Status 404 returned error can't find the container with id 42d0e47cad5ae3698247b91810078152673888ecdce7bb867e9375b1e77ede57 Sep 16 18:45:06.347906 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.347833 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:06.347906 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.347860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:06.348060 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.348000 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:06.350618 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.350595 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:45:06.350766 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.350602 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:45:06.351574 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.351555 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:45:06.351574 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.351564 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dgmmm\"" Sep 16 18:45:06.351574 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.351571 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:45:06.351777 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.351559 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pf72n\"" Sep 16 18:45:06.421305 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.421278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:06.421837 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:06.421396 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:06.421837 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:06.421453 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.421436899 +0000 UTC m=+37.586457233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:06.501882 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.501857 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="a4f6b9c72553c784169875047e5b7b29f0fc194c19512e945c4b7ec443b2a8ca" exitCode=0 Sep 16 18:45:06.501983 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.501940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"a4f6b9c72553c784169875047e5b7b29f0fc194c19512e945c4b7ec443b2a8ca"} Sep 16 18:45:06.503251 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:06.503228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k9wjj" event={"ID":"a470b680-0106-40e5-aab1-71f7f33a0598","Type":"ContainerStarted","Data":"42d0e47cad5ae3698247b91810078152673888ecdce7bb867e9375b1e77ede57"} Sep 16 18:45:07.509932 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:07.509892 2559 generic.go:356] "Generic (PLEG): container finished" podID="75eea085-6c20-456f-b234-ef793183e012" containerID="8906d25641e161617e8fbb62a071868c158d6be2736c3b41a82991314a7b3769" exitCode=0 Sep 16 18:45:07.510483 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:07.509944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerDied","Data":"8906d25641e161617e8fbb62a071868c158d6be2736c3b41a82991314a7b3769"} Sep 16 18:45:08.437745 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:08.437657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:08.437892 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:08.437844 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:08.437933 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:08.437926 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:12.437904391 +0000 UTC m=+41.602924723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:08.515621 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:08.515583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n2khs" event={"ID":"75eea085-6c20-456f-b234-ef793183e012","Type":"ContainerStarted","Data":"6b82002574bacc5f9dadfe7415ae7ea7a3490973a0efccc7487ac95859c513de"} Sep 16 18:45:08.517112 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:08.517090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k9wjj" event={"ID":"a470b680-0106-40e5-aab1-71f7f33a0598","Type":"ContainerStarted","Data":"3ac57de01c571045f3b671e4d3ac5fb845d58c284bab5511926ebe1e507fa3ab"} Sep 16 18:45:08.540651 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:08.540600 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n2khs" podStartSLOduration=8.969150763 podStartE2EDuration="37.540581977s" totalImagesPullingTime="28.571431214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.539346967 +0000 UTC m=+37.704367310" watchObservedRunningTime="2026-09-16 18:45:08.540581977 +0000 UTC m=+37.705602320" Sep 16 18:45:08.554951 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:08.554911 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k9wjj" podStartSLOduration=2.47299456 podStartE2EDuration="4.554898451s" totalImagesPullingTime="2.081903891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.554634241 +0000 UTC m=+37.719654585" watchObservedRunningTime="2026-09-16 18:45:08.554898451 +0000 UTC m=+37.719918790" Sep 16 18:45:12.463311 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:12.463272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:12.463703 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:12.463398 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:12.463703 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:12.463447 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.463434819 +0000 UTC m=+49.628455140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:20.519257 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:20.519215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:20.519685 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:20.519378 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:20.519685 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:20.519454 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.519437952 +0000 UTC m=+65.684458289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:30.496932 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:30.496898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhx6h" Sep 16 18:45:36.532910 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:36.532874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:45:36.533388 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:36.532994 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:36.533388 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:36.533047 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.533034564 +0000 UTC m=+97.698054898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:45:37.036233 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.036196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:45:37.039044 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.039023 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:45:37.047179 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:37.047162 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:45:37.047249 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:45:37.047212 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs podName:0c6db619-fa2b-4eae-9e3e-1b94fc47bba3 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:41.047197638 +0000 UTC m=+130.212217958 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs") pod "network-metrics-daemon-n7qvp" (UID: "0c6db619-fa2b-4eae-9e3e-1b94fc47bba3") : secret "metrics-daemon-secret" not found Sep 16 18:45:37.137547 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.137518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:37.137644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.137553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:37.140766 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.140745 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:45:37.140855 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.140784 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:45:37.150527 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.150505 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:45:37.151007 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.150989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84b69c03-718b-41c6-be39-cbc758435e44-original-pull-secret\") pod \"global-pull-secret-syncer-v9fdf\" (UID: \"84b69c03-718b-41c6-be39-cbc758435e44\") " pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:37.160427 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.160408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zst2\" (UniqueName: \"kubernetes.io/projected/3b9dd24b-6c17-4242-9237-c9a757140681-kube-api-access-9zst2\") pod \"network-check-target-vpt6j\" (UID: \"3b9dd24b-6c17-4242-9237-c9a757140681\") " pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:37.257810 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.257785 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9fdf" Sep 16 18:45:37.265490 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.265471 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pf72n\"" Sep 16 18:45:37.273951 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.273932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:37.557760 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.557611 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9fdf"] Sep 16 18:45:37.560465 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:45:37.560434 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b69c03_718b_41c6_be39_cbc758435e44.slice/crio-1a14dc106f403f8120f3ac8ba5c7a64f52d6d339cf7e790e4e9015ca63b3beba WatchSource:0}: Error finding container 1a14dc106f403f8120f3ac8ba5c7a64f52d6d339cf7e790e4e9015ca63b3beba: Status 404 returned error can't find the container with id 1a14dc106f403f8120f3ac8ba5c7a64f52d6d339cf7e790e4e9015ca63b3beba Sep 16 18:45:37.583300 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.583193 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vpt6j"] Sep 16 18:45:37.585367 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:37.585342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9fdf" event={"ID":"84b69c03-718b-41c6-be39-cbc758435e44","Type":"ContainerStarted","Data":"1a14dc106f403f8120f3ac8ba5c7a64f52d6d339cf7e790e4e9015ca63b3beba"} Sep 16 18:45:37.587369 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:45:37.587345 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9dd24b_6c17_4242_9237_c9a757140681.slice/crio-ba1050754a447e075afc9a7188bafdcced6700b21e59d34c3dce0840e9bdc45d WatchSource:0}: Error finding container ba1050754a447e075afc9a7188bafdcced6700b21e59d34c3dce0840e9bdc45d: Status 404 returned error can't find the container with id ba1050754a447e075afc9a7188bafdcced6700b21e59d34c3dce0840e9bdc45d Sep 16 18:45:38.590798 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:38.590756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vpt6j" event={"ID":"3b9dd24b-6c17-4242-9237-c9a757140681","Type":"ContainerStarted","Data":"ba1050754a447e075afc9a7188bafdcced6700b21e59d34c3dce0840e9bdc45d"} Sep 16 18:45:42.603198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:42.603164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9fdf" event={"ID":"84b69c03-718b-41c6-be39-cbc758435e44","Type":"ContainerStarted","Data":"1137c0797f0361235289686802fd1934f5dea6a7613560422f9ca242338cbec4"} Sep 16 18:45:42.604750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:42.604729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vpt6j" event={"ID":"3b9dd24b-6c17-4242-9237-c9a757140681","Type":"ContainerStarted","Data":"a4c7bdf0ef1da6e9576a3c8ab60ad963bf43030e8469b93be407e590c96d3ab8"} Sep 16 18:45:42.604862 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:42.604847 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:45:42.623518 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:42.623474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v9fdf" podStartSLOduration=67.218787961 podStartE2EDuration="1m11.623461429s" totalImagesPullingTime="4.404673468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.623071085 +0000 UTC m=+71.788091428" watchObservedRunningTime="2026-09-16 18:45:42.623461429 +0000 UTC m=+71.788481770" Sep 16 18:45:42.639407 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:45:42.639369 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vpt6j" podStartSLOduration=67.266021283 podStartE2EDuration="1m11.639357406s" totalImagesPullingTime="4.373336123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.638737078 +0000 UTC m=+71.803757421" watchObservedRunningTime="2026-09-16 18:45:42.639357406 +0000 UTC m=+71.804377748" Sep 16 18:46:08.544743 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:08.544711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:46:08.545142 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:08.544835 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:08.545142 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:08.544900 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls podName:772f55b7-eecf-4c9a-93a6-547a73a0b147 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:12.544887462 +0000 UTC m=+161.709907782 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls") pod "dns-default-w22ll" (UID: "772f55b7-eecf-4c9a-93a6-547a73a0b147") : secret "dns-default-metrics-tls" not found Sep 16 18:46:11.569230 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.569195 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp"] Sep 16 18:46:11.610517 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.610458 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ts4c"] Sep 16 18:46:11.610517 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.610478 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.613592 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.613570 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:46:11.613782 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.613757 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 18:46:11.613904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.613801 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 18:46:11.613904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.613820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 18:46:11.614230 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.614216 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g2vgw\"" Sep 16 18:46:11.630026 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.630009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp"] Sep 16 18:46:11.630119 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.630034 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ts4c"] Sep 16 18:46:11.630119 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.630095 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.633731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.633711 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 18:46:11.633833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.633743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s2xrf\"" Sep 16 18:46:11.633833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.633778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 18:46:11.633833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.633745 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 18:46:11.634042 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.634025 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:46:11.640137 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.640116 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 18:46:11.664791 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tn6\" (UniqueName: \"kubernetes.io/projected/f2064f60-12eb-4766-bf70-d91ed2f5845c-kube-api-access-m5tn6\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.664891 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-trusted-ca\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.664891 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.664961 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.664961 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przrw\" (UniqueName: \"kubernetes.io/projected/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-kube-api-access-przrw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.664961 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664928 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-config\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.664961 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.664949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2064f60-12eb-4766-bf70-d91ed2f5845c-serving-cert\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.686155 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.686133 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq"] Sep 16 18:46:11.702300 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.702280 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.705562 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.705543 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 18:46:11.705644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.705552 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:46:11.705644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.705619 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:46:11.707042 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.707019 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6f5zz\"" Sep 16 18:46:11.707150 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.707019 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 18:46:11.709714 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.709691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq"] Sep 16 18:46:11.765734 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.765859 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przrw\" (UniqueName: \"kubernetes.io/projected/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-kube-api-access-przrw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.765859 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.765859 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-config\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.766018 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2064f60-12eb-4766-bf70-d91ed2f5845c-serving-cert\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.766018 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.765978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tn6\" (UniqueName: \"kubernetes.io/projected/f2064f60-12eb-4766-bf70-d91ed2f5845c-kube-api-access-m5tn6\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.766190 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjsh\" (UniqueName: \"kubernetes.io/projected/20a4c8a4-63bb-436b-bb44-a26b8f81a746-kube-api-access-xpjsh\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.766190 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-trusted-ca\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.766190 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20a4c8a4-63bb-436b-bb44-a26b8f81a746-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.766190 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.766600 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-config\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.766687 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.766668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.767398 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.767383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2064f60-12eb-4766-bf70-d91ed2f5845c-trusted-ca\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.768134 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.768117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.768204 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.768145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2064f60-12eb-4766-bf70-d91ed2f5845c-serving-cert\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.782327 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.782291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-przrw\" (UniqueName: \"kubernetes.io/projected/ae82780d-8d4b-43d4-a9f7-a2926d86ba1b-kube-api-access-przrw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lx4jp\" (UID: \"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.784054 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.784033 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tn6\" (UniqueName: \"kubernetes.io/projected/f2064f60-12eb-4766-bf70-d91ed2f5845c-kube-api-access-m5tn6\") pod \"console-operator-696884fb9c-7ts4c\" (UID: \"f2064f60-12eb-4766-bf70-d91ed2f5845c\") " pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:11.867158 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.867094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.867158 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.867154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjsh\" (UniqueName: \"kubernetes.io/projected/20a4c8a4-63bb-436b-bb44-a26b8f81a746-kube-api-access-xpjsh\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.867279 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.867172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20a4c8a4-63bb-436b-bb44-a26b8f81a746-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.867279 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:11.867218 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:11.867375 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:11.867291 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls podName:20a4c8a4-63bb-436b-bb44-a26b8f81a746 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:12.36727485 +0000 UTC m=+101.532295175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zh4wq" (UID: "20a4c8a4-63bb-436b-bb44-a26b8f81a746") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:11.867719 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.867702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20a4c8a4-63bb-436b-bb44-a26b8f81a746-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.875904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.875884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjsh\" (UniqueName: \"kubernetes.io/projected/20a4c8a4-63bb-436b-bb44-a26b8f81a746-kube-api-access-xpjsh\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:11.919173 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.919152 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" Sep 16 18:46:11.938351 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:11.938334 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:12.217392 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:12.217294 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp"] Sep 16 18:46:12.220639 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:12.220599 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae82780d_8d4b_43d4_a9f7_a2926d86ba1b.slice/crio-4d6791efdb5831859aa2cb39186878a1f22995b3354c5afcb9c5db3930b3173c WatchSource:0}: Error finding container 4d6791efdb5831859aa2cb39186878a1f22995b3354c5afcb9c5db3930b3173c: Status 404 returned error can't find the container with id 4d6791efdb5831859aa2cb39186878a1f22995b3354c5afcb9c5db3930b3173c Sep 16 18:46:12.245507 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:12.245482 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ts4c"] Sep 16 18:46:12.248115 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:12.248089 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2064f60_12eb_4766_bf70_d91ed2f5845c.slice/crio-0ad392e8a025b967bbe54a5560855b882c16f00905ec9d3d9a6372cd2ddc69a4 WatchSource:0}: Error finding container 0ad392e8a025b967bbe54a5560855b882c16f00905ec9d3d9a6372cd2ddc69a4: Status 404 returned error can't find the container with id 0ad392e8a025b967bbe54a5560855b882c16f00905ec9d3d9a6372cd2ddc69a4 Sep 16 18:46:12.371768 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:12.371740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:12.371941 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:12.371855 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:12.371941 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:12.371914 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls podName:20a4c8a4-63bb-436b-bb44-a26b8f81a746 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.37189687 +0000 UTC m=+102.536917207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zh4wq" (UID: "20a4c8a4-63bb-436b-bb44-a26b8f81a746") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:12.678657 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:12.678620 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" event={"ID":"f2064f60-12eb-4766-bf70-d91ed2f5845c","Type":"ContainerStarted","Data":"0ad392e8a025b967bbe54a5560855b882c16f00905ec9d3d9a6372cd2ddc69a4"} Sep 16 18:46:12.679736 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:12.679714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" event={"ID":"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b","Type":"ContainerStarted","Data":"4d6791efdb5831859aa2cb39186878a1f22995b3354c5afcb9c5db3930b3173c"} Sep 16 18:46:13.378391 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.378359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:13.378683 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:13.378520 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:13.378683 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:13.378598 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls podName:20a4c8a4-63bb-436b-bb44-a26b8f81a746 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.378576667 +0000 UTC m=+104.543597001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zh4wq" (UID: "20a4c8a4-63bb-436b-bb44-a26b8f81a746") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:13.610534 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.610504 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vpt6j" Sep 16 18:46:13.749608 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.749534 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96jtp"] Sep 16 18:46:13.773602 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.773571 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96jtp"] Sep 16 18:46:13.773751 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.773683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" Sep 16 18:46:13.777127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.777104 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f6g66\"" Sep 16 18:46:13.882709 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.882673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xm4\" (UniqueName: \"kubernetes.io/projected/fa6f9a4a-929c-4bd0-9df5-7a331970122d-kube-api-access-29xm4\") pod \"network-check-source-6f8b768455-96jtp\" (UID: \"fa6f9a4a-929c-4bd0-9df5-7a331970122d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" Sep 16 18:46:13.984118 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.984085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xm4\" (UniqueName: \"kubernetes.io/projected/fa6f9a4a-929c-4bd0-9df5-7a331970122d-kube-api-access-29xm4\") pod \"network-check-source-6f8b768455-96jtp\" (UID: \"fa6f9a4a-929c-4bd0-9df5-7a331970122d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" Sep 16 18:46:13.995701 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:13.995658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xm4\" (UniqueName: \"kubernetes.io/projected/fa6f9a4a-929c-4bd0-9df5-7a331970122d-kube-api-access-29xm4\") pod \"network-check-source-6f8b768455-96jtp\" (UID: \"fa6f9a4a-929c-4bd0-9df5-7a331970122d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" Sep 16 18:46:14.085133 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:14.085088 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" Sep 16 18:46:14.938777 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:14.938713 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96jtp"] Sep 16 18:46:14.941654 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:14.941613 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6f9a4a_929c_4bd0_9df5_7a331970122d.slice/crio-7cddb6993f409f353b7b43dbf73223cfd411734bc50bb692a528b6c6f5cf1aac WatchSource:0}: Error finding container 7cddb6993f409f353b7b43dbf73223cfd411734bc50bb692a528b6c6f5cf1aac: Status 404 returned error can't find the container with id 7cddb6993f409f353b7b43dbf73223cfd411734bc50bb692a528b6c6f5cf1aac Sep 16 18:46:15.397421 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.397378 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:15.397589 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:15.397512 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:15.397589 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:15.397576 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls podName:20a4c8a4-63bb-436b-bb44-a26b8f81a746 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.397557763 +0000 UTC m=+108.562578100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zh4wq" (UID: "20a4c8a4-63bb-436b-bb44-a26b8f81a746") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:15.690693 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.690620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/0.log" Sep 16 18:46:15.690693 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.690670 2559 generic.go:356] "Generic (PLEG): container finished" podID="f2064f60-12eb-4766-bf70-d91ed2f5845c" containerID="a34aa6ef437a8258d8f98fb30649de82827e4de52ef92734c80f82164647f8fa" exitCode=255 Sep 16 18:46:15.690904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.690759 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" event={"ID":"f2064f60-12eb-4766-bf70-d91ed2f5845c","Type":"ContainerDied","Data":"a34aa6ef437a8258d8f98fb30649de82827e4de52ef92734c80f82164647f8fa"} Sep 16 18:46:15.690987 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.690971 2559 scope.go:117] "RemoveContainer" containerID="a34aa6ef437a8258d8f98fb30649de82827e4de52ef92734c80f82164647f8fa" Sep 16 18:46:15.692459 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.692433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" event={"ID":"fa6f9a4a-929c-4bd0-9df5-7a331970122d","Type":"ContainerStarted","Data":"f0eb8590b62552bb4f4798a926ea37ab12e8cc167291cc173c9412f32d3e61f9"} Sep 16 18:46:15.692578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.692484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" event={"ID":"fa6f9a4a-929c-4bd0-9df5-7a331970122d","Type":"ContainerStarted","Data":"7cddb6993f409f353b7b43dbf73223cfd411734bc50bb692a528b6c6f5cf1aac"} Sep 16 18:46:15.694075 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.694052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" event={"ID":"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b","Type":"ContainerStarted","Data":"18871211d56d4726c9c20636bc18749e85d9a950e9e9029b84fdf2e10e8fd171"} Sep 16 18:46:15.728424 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.728369 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" podStartSLOduration=2.275765243 podStartE2EDuration="4.728351277s" totalImagesPullingTime="2.452586034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:15.72814033 +0000 UTC m=+104.893160673" watchObservedRunningTime="2026-09-16 18:46:15.728351277 +0000 UTC m=+104.893371620" Sep 16 18:46:15.744464 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.744419 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96jtp" podStartSLOduration=2.744380149 podStartE2EDuration="2.744380149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:15.743591386 +0000 UTC m=+104.908611729" watchObservedRunningTime="2026-09-16 18:46:15.744380149 +0000 UTC m=+104.909400492" Sep 16 18:46:15.877759 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.877725 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r7fvd"] Sep 16 18:46:15.880843 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.880818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:15.884207 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.884183 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:46:15.884386 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.884364 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6l7cw\"" Sep 16 18:46:15.885079 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.885062 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:46:15.885352 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.885336 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:46:15.885413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.885369 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:46:15.904928 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:15.904903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r7fvd"] Sep 16 18:46:16.003582 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.003493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.003582 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.003528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gksxl\" (UniqueName: \"kubernetes.io/projected/16f87351-5974-49af-86f4-b91e72da6b85-kube-api-access-gksxl\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.003582 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.003553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.004110 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.003608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16f87351-5974-49af-86f4-b91e72da6b85-data-volume\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.004110 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.003690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16f87351-5974-49af-86f4-b91e72da6b85-crio-socket\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.104847 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.104807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16f87351-5974-49af-86f4-b91e72da6b85-data-volume\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105015 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.104879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16f87351-5974-49af-86f4-b91e72da6b85-crio-socket\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105015 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.104914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105015 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.104929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gksxl\" (UniqueName: \"kubernetes.io/projected/16f87351-5974-49af-86f4-b91e72da6b85-kube-api-access-gksxl\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105015 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.104947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.105008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16f87351-5974-49af-86f4-b91e72da6b85-crio-socket\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.105029 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.105049 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.105108 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.605086578 +0000 UTC m=+105.770106911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.105126 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.605114398 +0000 UTC m=+105.770134747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : configmap "kube-rbac-proxy" not found Sep 16 18:46:16.105211 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.105183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16f87351-5974-49af-86f4-b91e72da6b85-data-volume\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.132555 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.132530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gksxl\" (UniqueName: \"kubernetes.io/projected/16f87351-5974-49af-86f4-b91e72da6b85-kube-api-access-gksxl\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.609091 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.609050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.609091 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.609094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:16.609294 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.609229 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.609349 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.609310 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.609294729 +0000 UTC m=+106.774315067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.609349 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.609311 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:16.609423 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.609412 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.609393555 +0000 UTC m=+106.774413887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : configmap "kube-rbac-proxy" not found Sep 16 18:46:16.699152 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699125 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:46:16.699566 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699550 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/0.log" Sep 16 18:46:16.699641 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699584 2559 generic.go:356] "Generic (PLEG): container finished" podID="f2064f60-12eb-4766-bf70-d91ed2f5845c" containerID="2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f" exitCode=255 Sep 16 18:46:16.699641 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" event={"ID":"f2064f60-12eb-4766-bf70-d91ed2f5845c","Type":"ContainerDied","Data":"2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f"} Sep 16 18:46:16.699742 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699678 2559 scope.go:117] "RemoveContainer" containerID="a34aa6ef437a8258d8f98fb30649de82827e4de52ef92734c80f82164647f8fa" Sep 16 18:46:16.699959 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.699936 2559 scope.go:117] "RemoveContainer" containerID="2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f" Sep 16 18:46:16.700133 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:16.700115 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ts4c_openshift-console-operator(f2064f60-12eb-4766-bf70-d91ed2f5845c)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" podUID="f2064f60-12eb-4766-bf70-d91ed2f5845c" Sep 16 18:46:16.910876 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.910804 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps"] Sep 16 18:46:16.914993 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.914972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" Sep 16 18:46:16.919630 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.919611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mz4sk\"" Sep 16 18:46:16.919966 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.919947 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 18:46:16.920047 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.919949 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 18:46:16.928471 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:16.928451 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps"] Sep 16 18:46:17.011247 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.011213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8vm\" (UniqueName: \"kubernetes.io/projected/f3345352-c4b7-4b44-9372-3e5c21275e17-kube-api-access-mx8vm\") pod \"migrator-8b4d697c5-kh8ps\" (UID: \"f3345352-c4b7-4b44-9372-3e5c21275e17\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" Sep 16 18:46:17.111798 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.111758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8vm\" (UniqueName: \"kubernetes.io/projected/f3345352-c4b7-4b44-9372-3e5c21275e17-kube-api-access-mx8vm\") pod \"migrator-8b4d697c5-kh8ps\" (UID: \"f3345352-c4b7-4b44-9372-3e5c21275e17\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" Sep 16 18:46:17.121670 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.121644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8vm\" (UniqueName: \"kubernetes.io/projected/f3345352-c4b7-4b44-9372-3e5c21275e17-kube-api-access-mx8vm\") pod \"migrator-8b4d697c5-kh8ps\" (UID: \"f3345352-c4b7-4b44-9372-3e5c21275e17\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" Sep 16 18:46:17.223790 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.223729 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" Sep 16 18:46:17.489542 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.489476 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps"] Sep 16 18:46:17.492199 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:17.492179 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3345352_c4b7_4b44_9372_3e5c21275e17.slice/crio-7d843d6a8abd46f8709846bf1c52e14f368129152227a9350d4f43b6ed2bb455 WatchSource:0}: Error finding container 7d843d6a8abd46f8709846bf1c52e14f368129152227a9350d4f43b6ed2bb455: Status 404 returned error can't find the container with id 7d843d6a8abd46f8709846bf1c52e14f368129152227a9350d4f43b6ed2bb455 Sep 16 18:46:17.615231 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.615197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:17.615414 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.615236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:17.615414 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:17.615390 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:17.615491 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:17.615460 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.615445746 +0000 UTC m=+108.780466083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:17.615491 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:17.615390 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:17.615565 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:17.615505 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.615497153 +0000 UTC m=+108.780517488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : configmap "kube-rbac-proxy" not found Sep 16 18:46:17.628416 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.628394 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2qwhj_14e13b94-9f75-43d9-a98b-8899a3926385/dns-node-resolver/0.log" Sep 16 18:46:17.705130 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.705108 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:46:17.705487 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.705471 2559 scope.go:117] "RemoveContainer" containerID="2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f" Sep 16 18:46:17.705664 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:17.705644 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ts4c_openshift-console-operator(f2064f60-12eb-4766-bf70-d91ed2f5845c)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" podUID="f2064f60-12eb-4766-bf70-d91ed2f5845c" Sep 16 18:46:17.706496 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:17.706473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" event={"ID":"f3345352-c4b7-4b44-9372-3e5c21275e17","Type":"ContainerStarted","Data":"7d843d6a8abd46f8709846bf1c52e14f368129152227a9350d4f43b6ed2bb455"} Sep 16 18:46:18.757154 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.757116 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9"] Sep 16 18:46:18.760607 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.760578 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:18.763710 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.763689 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 18:46:18.764169 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.764149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fw5hm\"" Sep 16 18:46:18.764554 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.764537 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 18:46:18.771431 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.771411 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9"] Sep 16 18:46:18.825111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.825030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:18.825111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.825068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:18.839424 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.839403 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rk528_c624e892-e030-4e19-a78c-c9a24b6abc1b/node-ca/0.log" Sep 16 18:46:18.925761 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.925735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:18.925874 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.925796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:18.925972 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:18.925956 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:46:18.926038 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:18.926026 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert podName:fa283c3d-3f28-4add-9c56-59cf6c6c6bfc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.426005316 +0000 UTC m=+108.591025653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rkvs9" (UID: "fa283c3d-3f28-4add-9c56-59cf6c6c6bfc") : secret "networking-console-plugin-cert" not found Sep 16 18:46:18.926424 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:18.926407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:19.430654 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.430625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:19.430841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.430694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:19.430841 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.430770 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:19.430841 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.430788 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:46:19.430841 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.430834 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls podName:20a4c8a4-63bb-436b-bb44-a26b8f81a746 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:27.430818205 +0000 UTC m=+116.595838530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zh4wq" (UID: "20a4c8a4-63bb-436b-bb44-a26b8f81a746") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:46:19.430985 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.430849 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert podName:fa283c3d-3f28-4add-9c56-59cf6c6c6bfc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.430842912 +0000 UTC m=+109.595863232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rkvs9" (UID: "fa283c3d-3f28-4add-9c56-59cf6c6c6bfc") : secret "networking-console-plugin-cert" not found Sep 16 18:46:19.629083 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.629048 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k9wjj_a470b680-0106-40e5-aab1-71f7f33a0598/serve-healthcheck-canary/0.log" Sep 16 18:46:19.633183 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.633163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:19.633235 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.633196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:19.633308 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.633293 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:19.633365 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.633355 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.633343345 +0000 UTC m=+112.798363670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:19.633436 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.633299 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:19.633436 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:19.633418 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.633406124 +0000 UTC m=+112.798426444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : configmap "kube-rbac-proxy" not found Sep 16 18:46:19.714027 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.713954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" event={"ID":"f3345352-c4b7-4b44-9372-3e5c21275e17","Type":"ContainerStarted","Data":"157d930ea2b7370bf3ee7c12208db84a7dcce5e4dd044c91e9ea92e577e618fe"} Sep 16 18:46:19.714027 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.714011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" event={"ID":"f3345352-c4b7-4b44-9372-3e5c21275e17","Type":"ContainerStarted","Data":"9d0975c01b1d2bce12aef62d48e681d710560379fbf3617a89d985eb6e322edf"} Sep 16 18:46:19.735589 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:19.735548 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kh8ps" podStartSLOduration=2.410593417 podStartE2EDuration="3.735533753s" totalImagesPullingTime="1.324940336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:19.73512621 +0000 UTC m=+108.900146552" watchObservedRunningTime="2026-09-16 18:46:19.735533753 +0000 UTC m=+108.900554143" Sep 16 18:46:20.439693 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:20.439644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:20.440153 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:20.439778 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:46:20.440153 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:20.439851 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert podName:fa283c3d-3f28-4add-9c56-59cf6c6c6bfc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.439830208 +0000 UTC m=+111.604850535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rkvs9" (UID: "fa283c3d-3f28-4add-9c56-59cf6c6c6bfc") : secret "networking-console-plugin-cert" not found Sep 16 18:46:21.939485 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:21.939444 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:21.939870 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:21.939534 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:21.939870 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:21.939839 2559 scope.go:117] "RemoveContainer" containerID="2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f" Sep 16 18:46:21.939995 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:21.939974 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ts4c_openshift-console-operator(f2064f60-12eb-4766-bf70-d91ed2f5845c)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" podUID="f2064f60-12eb-4766-bf70-d91ed2f5845c" Sep 16 18:46:22.453099 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:22.453065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:22.453292 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:22.453196 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:46:22.453292 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:22.453261 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert podName:fa283c3d-3f28-4add-9c56-59cf6c6c6bfc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:26.453243632 +0000 UTC m=+115.618263971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rkvs9" (UID: "fa283c3d-3f28-4add-9c56-59cf6c6c6bfc") : secret "networking-console-plugin-cert" not found Sep 16 18:46:23.662568 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:23.662534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:23.662568 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:23.662573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:23.663043 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:23.662672 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:23.663043 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:23.662689 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:23.663043 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:23.662729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:31.662712398 +0000 UTC m=+120.827732741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : configmap "kube-rbac-proxy" not found Sep 16 18:46:23.663043 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:23.662754 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls podName:16f87351-5974-49af-86f4-b91e72da6b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:31.662735359 +0000 UTC m=+120.827755686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r7fvd" (UID: "16f87351-5974-49af-86f4-b91e72da6b85") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:26.484185 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:26.484140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:26.484580 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:26.484292 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:46:26.484580 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:46:26.484376 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert podName:fa283c3d-3f28-4add-9c56-59cf6c6c6bfc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:34.484361485 +0000 UTC m=+123.649381809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rkvs9" (UID: "fa283c3d-3f28-4add-9c56-59cf6c6c6bfc") : secret "networking-console-plugin-cert" not found Sep 16 18:46:27.491848 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:27.491819 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:27.494117 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:27.494095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a4c8a4-63bb-436b-bb44-a26b8f81a746-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zh4wq\" (UID: \"20a4c8a4-63bb-436b-bb44-a26b8f81a746\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:27.611001 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:27.610971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" Sep 16 18:46:27.881914 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:27.881875 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq"] Sep 16 18:46:27.884907 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:27.884884 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a4c8a4_63bb_436b_bb44_a26b8f81a746.slice/crio-1116ce2ae3387b9a7cf48f5fc5b81880f18c7bad8e2f4844312e55ee22bec0b9 WatchSource:0}: Error finding container 1116ce2ae3387b9a7cf48f5fc5b81880f18c7bad8e2f4844312e55ee22bec0b9: Status 404 returned error can't find the container with id 1116ce2ae3387b9a7cf48f5fc5b81880f18c7bad8e2f4844312e55ee22bec0b9 Sep 16 18:46:28.743096 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:28.743055 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" event={"ID":"20a4c8a4-63bb-436b-bb44-a26b8f81a746","Type":"ContainerStarted","Data":"1116ce2ae3387b9a7cf48f5fc5b81880f18c7bad8e2f4844312e55ee22bec0b9"} Sep 16 18:46:29.747211 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:29.747174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" event={"ID":"20a4c8a4-63bb-436b-bb44-a26b8f81a746","Type":"ContainerStarted","Data":"3d3806fa239926a9fe7aa266f5e0552c18273e38213e7ff1ef43d6363aa66570"} Sep 16 18:46:29.769239 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:29.769184 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zh4wq" podStartSLOduration=17.133397242 podStartE2EDuration="18.769170653s" totalImagesPullingTime="1.635773411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:29.768029658 +0000 UTC m=+118.933050029" watchObservedRunningTime="2026-09-16 18:46:29.769170653 +0000 UTC m=+118.934190992" Sep 16 18:46:31.726943 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.726910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:31.727302 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.726950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:31.727430 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.727413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16f87351-5974-49af-86f4-b91e72da6b85-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:31.729082 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.729063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16f87351-5974-49af-86f4-b91e72da6b85-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r7fvd\" (UID: \"16f87351-5974-49af-86f4-b91e72da6b85\") " pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:31.791804 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.791776 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6l7cw\"" Sep 16 18:46:31.800020 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:31.800001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r7fvd" Sep 16 18:46:32.079153 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:32.079125 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r7fvd"] Sep 16 18:46:32.082282 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:32.082255 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f87351_5974_49af_86f4_b91e72da6b85.slice/crio-a53ce4fa0aaccb5a191289676bfa8a7d36407ef24121a6328de9b08d6b93d34b WatchSource:0}: Error finding container a53ce4fa0aaccb5a191289676bfa8a7d36407ef24121a6328de9b08d6b93d34b: Status 404 returned error can't find the container with id a53ce4fa0aaccb5a191289676bfa8a7d36407ef24121a6328de9b08d6b93d34b Sep 16 18:46:32.758095 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:32.758064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7fvd" event={"ID":"16f87351-5974-49af-86f4-b91e72da6b85","Type":"ContainerStarted","Data":"a5cd61f247852f39b158d2dcc9f559d0c6ca9c3deb861fac9d579ca68d830687"} Sep 16 18:46:32.758482 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:32.758118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7fvd" event={"ID":"16f87351-5974-49af-86f4-b91e72da6b85","Type":"ContainerStarted","Data":"a53ce4fa0aaccb5a191289676bfa8a7d36407ef24121a6328de9b08d6b93d34b"} Sep 16 18:46:33.763734 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:33.763698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7fvd" event={"ID":"16f87351-5974-49af-86f4-b91e72da6b85","Type":"ContainerStarted","Data":"862b53ee1472f167c347d965a8bc1a57cdeb08d7d34ba6425481aadd3bc6b3cc"} Sep 16 18:46:34.348922 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.348890 2559 scope.go:117] "RemoveContainer" containerID="2b866aefc897d60d35486d5ceb2493511bdb478241eed8df884685c04e1a422f" Sep 16 18:46:34.548215 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.548183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:34.550558 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.550535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa283c3d-3f28-4add-9c56-59cf6c6c6bfc-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rkvs9\" (UID: \"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:34.675558 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.675483 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fw5hm\"" Sep 16 18:46:34.682069 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.682049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" Sep 16 18:46:34.768874 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.768845 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:46:34.769257 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.768934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" event={"ID":"f2064f60-12eb-4766-bf70-d91ed2f5845c","Type":"ContainerStarted","Data":"201df1283d1470d1f0bbf985fed888dcfc78ccb500762b526253c84c3691c6b2"} Sep 16 18:46:34.769257 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.769209 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:34.771193 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.771171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r7fvd" event={"ID":"16f87351-5974-49af-86f4-b91e72da6b85","Type":"ContainerStarted","Data":"c2e84355c841a8375befc880b674a9e002b2df25497e8884b6b6e76b67c16268"} Sep 16 18:46:34.789623 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.789586 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" podStartSLOduration=21.359842374 podStartE2EDuration="23.789572028s" totalImagesPullingTime="2.429729654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:34.789299107 +0000 UTC m=+123.954319449" watchObservedRunningTime="2026-09-16 18:46:34.789572028 +0000 UTC m=+123.954592371" Sep 16 18:46:34.815724 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.815678 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r7fvd" podStartSLOduration=17.582144937 podStartE2EDuration="19.815666751s" totalImagesPullingTime="2.233521814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:34.814526979 +0000 UTC m=+123.979547323" watchObservedRunningTime="2026-09-16 18:46:34.815666751 +0000 UTC m=+123.980687334" Sep 16 18:46:34.828723 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.828702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-7ts4c" Sep 16 18:46:34.955696 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:34.955624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9"] Sep 16 18:46:34.959354 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:34.959310 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa283c3d_3f28_4add_9c56_59cf6c6c6bfc.slice/crio-c400ed0d951de03b39565adefc559d5c5cbea6185b51588510bd8d3737ab7d6d WatchSource:0}: Error finding container c400ed0d951de03b39565adefc559d5c5cbea6185b51588510bd8d3737ab7d6d: Status 404 returned error can't find the container with id c400ed0d951de03b39565adefc559d5c5cbea6185b51588510bd8d3737ab7d6d Sep 16 18:46:35.776489 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:35.776455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" event={"ID":"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc","Type":"ContainerStarted","Data":"c400ed0d951de03b39565adefc559d5c5cbea6185b51588510bd8d3737ab7d6d"} Sep 16 18:46:36.781624 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:36.781589 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" event={"ID":"fa283c3d-3f28-4add-9c56-59cf6c6c6bfc","Type":"ContainerStarted","Data":"7d770f946d6ccc48dcadded961b6af2383ee6166dfa733394708ea9b05321a01"} Sep 16 18:46:36.801392 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:36.801351 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rkvs9" podStartSLOduration=17.798497935 podStartE2EDuration="18.801316121s" totalImagesPullingTime="1.002818186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.80075609 +0000 UTC m=+125.965776432" watchObservedRunningTime="2026-09-16 18:46:36.801316121 +0000 UTC m=+125.966336482" Sep 16 18:46:41.101978 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.101938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:46:41.104176 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.104157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6db619-fa2b-4eae-9e3e-1b94fc47bba3-metrics-certs\") pod \"network-metrics-daemon-n7qvp\" (UID: \"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3\") " pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:46:41.169686 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.169657 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dgmmm\"" Sep 16 18:46:41.177556 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.177536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7qvp" Sep 16 18:46:41.444921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.444844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7qvp"] Sep 16 18:46:41.447460 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:41.447423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6db619_fa2b_4eae_9e3e_1b94fc47bba3.slice/crio-5408c7cf121d1b81f46acce0c431525be5a21ad297d8d04fd95d40388163fdb2 WatchSource:0}: Error finding container 5408c7cf121d1b81f46acce0c431525be5a21ad297d8d04fd95d40388163fdb2: Status 404 returned error can't find the container with id 5408c7cf121d1b81f46acce0c431525be5a21ad297d8d04fd95d40388163fdb2 Sep 16 18:46:41.797850 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:41.797818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7qvp" event={"ID":"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3","Type":"ContainerStarted","Data":"5408c7cf121d1b81f46acce0c431525be5a21ad297d8d04fd95d40388163fdb2"} Sep 16 18:46:42.802533 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:42.802503 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7qvp" event={"ID":"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3","Type":"ContainerStarted","Data":"2eafd2a730c9e88c799ce3615e570e4290cb05803af23372efba36a252bc264d"} Sep 16 18:46:42.802885 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:42.802553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7qvp" event={"ID":"0c6db619-fa2b-4eae-9e3e-1b94fc47bba3","Type":"ContainerStarted","Data":"f08d7fa7027f74049a5b99d6e043ca47b53246bc3a51612b396846a4da724a2c"} Sep 16 18:46:42.823688 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:42.823636 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n7qvp" podStartSLOduration=130.893370529 podStartE2EDuration="2m11.823621215s" totalImagesPullingTime="930.250686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.819591365 +0000 UTC m=+131.984611709" watchObservedRunningTime="2026-09-16 18:46:42.823621215 +0000 UTC m=+131.988641591" Sep 16 18:46:43.375355 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.375286 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-t6mt7"] Sep 16 18:46:43.378366 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.378347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:46:43.381599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.381579 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k82pn\"" Sep 16 18:46:43.381990 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.381970 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 18:46:43.385481 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.385463 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 18:46:43.398792 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.398772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t6mt7"] Sep 16 18:46:43.440720 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.440699 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-959db6b9-vbg5r"] Sep 16 18:46:43.443561 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.443547 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.445987 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.445962 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 18:46:43.446091 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.446017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 18:46:43.446140 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.446094 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kp4l\"" Sep 16 18:46:43.446242 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.446223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 18:46:43.458123 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.458103 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 18:46:43.475847 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.475819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-959db6b9-vbg5r"] Sep 16 18:46:43.476720 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.476701 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq"] Sep 16 18:46:43.480000 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.479984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:43.482623 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.482607 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z26gr\"" Sep 16 18:46:43.483001 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.482978 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 18:46:43.499990 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.499967 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq"] Sep 16 18:46:43.519825 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.519800 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqfh\" (UniqueName: \"kubernetes.io/projected/362e2e82-5781-437b-bc02-916e42f9e89f-kube-api-access-dfqfh\") pod \"downloads-76f99584f4-t6mt7\" (UID: \"362e2e82-5781-437b-bc02-916e42f9e89f\") " pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:46:43.620930 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.620896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-installation-pull-secrets\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.620938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-image-registry-private-configuration\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.620981 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-tls\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cae0d6-243d-4b85-8877-9b6f38ef3771-ca-trust-extracted\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eae3fe52-bd69-4c4a-94b9-3900483187d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dn7gq\" (UID: \"eae3fe52-bd69-4c4a-94b9-3900483187d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:43.621270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-certificates\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-trusted-ca\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-bound-sa-token\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfp6\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-kube-api-access-kzfp6\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.621270 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.621229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqfh\" (UniqueName: \"kubernetes.io/projected/362e2e82-5781-437b-bc02-916e42f9e89f-kube-api-access-dfqfh\") pod \"downloads-76f99584f4-t6mt7\" (UID: \"362e2e82-5781-437b-bc02-916e42f9e89f\") " pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:46:43.631543 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.631486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqfh\" (UniqueName: \"kubernetes.io/projected/362e2e82-5781-437b-bc02-916e42f9e89f-kube-api-access-dfqfh\") pod \"downloads-76f99584f4-t6mt7\" (UID: \"362e2e82-5781-437b-bc02-916e42f9e89f\") " pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:46:43.687506 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.687480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:46:43.722432 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cae0d6-243d-4b85-8877-9b6f38ef3771-ca-trust-extracted\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722571 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eae3fe52-bd69-4c4a-94b9-3900483187d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dn7gq\" (UID: \"eae3fe52-bd69-4c4a-94b9-3900483187d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:43.722648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-certificates\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722648 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-trusted-ca\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-bound-sa-token\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722750 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfp6\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-kube-api-access-kzfp6\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722850 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68cae0d6-243d-4b85-8877-9b6f38ef3771-ca-trust-extracted\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722947 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-installation-pull-secrets\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.722997 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.722970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-image-registry-private-configuration\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.723052 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.723025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-tls\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.723498 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.723457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-certificates\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.723650 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.723609 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68cae0d6-243d-4b85-8877-9b6f38ef3771-trusted-ca\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.725533 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.725489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eae3fe52-bd69-4c4a-94b9-3900483187d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dn7gq\" (UID: \"eae3fe52-bd69-4c4a-94b9-3900483187d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:43.725731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.725683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-installation-pull-secrets\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.725731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.725684 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68cae0d6-243d-4b85-8877-9b6f38ef3771-image-registry-private-configuration\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.726925 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.726905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-registry-tls\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.734568 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.734529 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-bound-sa-token\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.737101 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.736865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfp6\" (UniqueName: \"kubernetes.io/projected/68cae0d6-243d-4b85-8877-9b6f38ef3771-kube-api-access-kzfp6\") pod \"image-registry-959db6b9-vbg5r\" (UID: \"68cae0d6-243d-4b85-8877-9b6f38ef3771\") " pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.752357 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.752334 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:43.788837 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.788804 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:43.972019 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:43.971945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t6mt7"] Sep 16 18:46:43.974603 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:43.974579 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362e2e82_5781_437b_bc02_916e42f9e89f.slice/crio-f4d84b94110c6ffa9ec5c9bde1ef9cfd2abab69b720a219df976d47dfd80d83c WatchSource:0}: Error finding container f4d84b94110c6ffa9ec5c9bde1ef9cfd2abab69b720a219df976d47dfd80d83c: Status 404 returned error can't find the container with id f4d84b94110c6ffa9ec5c9bde1ef9cfd2abab69b720a219df976d47dfd80d83c Sep 16 18:46:44.037255 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.037135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-959db6b9-vbg5r"] Sep 16 18:46:44.039736 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:44.039707 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cae0d6_243d_4b85_8877_9b6f38ef3771.slice/crio-a346048b67734fc24427766babbd2bb1a22976fd6d79ac81bb29a1c5b0b3694f WatchSource:0}: Error finding container a346048b67734fc24427766babbd2bb1a22976fd6d79ac81bb29a1c5b0b3694f: Status 404 returned error can't find the container with id a346048b67734fc24427766babbd2bb1a22976fd6d79ac81bb29a1c5b0b3694f Sep 16 18:46:44.065941 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.065915 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq"] Sep 16 18:46:44.071236 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:44.071215 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae3fe52_bd69_4c4a_94b9_3900483187d8.slice/crio-25e251176479359991d90742f003542316fa11d0ed125ebdbb55853e83003880 WatchSource:0}: Error finding container 25e251176479359991d90742f003542316fa11d0ed125ebdbb55853e83003880: Status 404 returned error can't find the container with id 25e251176479359991d90742f003542316fa11d0ed125ebdbb55853e83003880 Sep 16 18:46:44.813090 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.813052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" event={"ID":"68cae0d6-243d-4b85-8877-9b6f38ef3771","Type":"ContainerStarted","Data":"8eaeb3b3dc0640471d92bec98698149d414e3a616a862683ae570a6f002be09e"} Sep 16 18:46:44.813303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.813116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" event={"ID":"68cae0d6-243d-4b85-8877-9b6f38ef3771","Type":"ContainerStarted","Data":"a346048b67734fc24427766babbd2bb1a22976fd6d79ac81bb29a1c5b0b3694f"} Sep 16 18:46:44.813303 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.813179 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:46:44.814505 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.814475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t6mt7" event={"ID":"362e2e82-5781-437b-bc02-916e42f9e89f","Type":"ContainerStarted","Data":"f4d84b94110c6ffa9ec5c9bde1ef9cfd2abab69b720a219df976d47dfd80d83c"} Sep 16 18:46:44.815801 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.815769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" event={"ID":"eae3fe52-bd69-4c4a-94b9-3900483187d8","Type":"ContainerStarted","Data":"25e251176479359991d90742f003542316fa11d0ed125ebdbb55853e83003880"} Sep 16 18:46:44.835843 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:44.835727 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" podStartSLOduration=1.835708299 podStartE2EDuration="1.835708299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:44.835045565 +0000 UTC m=+134.000065908" watchObservedRunningTime="2026-09-16 18:46:44.835708299 +0000 UTC m=+134.000728641" Sep 16 18:46:45.821198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:45.821147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" event={"ID":"eae3fe52-bd69-4c4a-94b9-3900483187d8","Type":"ContainerStarted","Data":"31683641215ebbd630e4f8b6d22b373cc0f20a2ec4817fc5e8e4beb8d4b473c3"} Sep 16 18:46:45.821696 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:45.821660 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:45.827461 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:45.827438 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" Sep 16 18:46:45.838293 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:45.838244 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dn7gq" podStartSLOduration=1.7874464209999998 podStartE2EDuration="2.838229629s" totalImagesPullingTime="1.050783208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:45.837725059 +0000 UTC m=+135.002745403" watchObservedRunningTime="2026-09-16 18:46:45.838229629 +0000 UTC m=+135.003249961" Sep 16 18:46:46.112426 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.112334 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:46:46.115300 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.115279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.117987 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.117964 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 18:46:46.118101 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.118014 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 18:46:46.118101 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.117964 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 18:46:46.118218 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.118190 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 18:46:46.118854 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.118833 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mqltq\"" Sep 16 18:46:46.118965 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.118856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 18:46:46.129038 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.129013 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:46:46.143499 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.143477 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn"] Sep 16 18:46:46.145809 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.145790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.148579 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.148558 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 18:46:46.148683 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.148560 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:46:46.148969 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.148912 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 18:46:46.149104 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.148932 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-v9c6k\"" Sep 16 18:46:46.156220 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.156195 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn"] Sep 16 18:46:46.247362 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xj4d\" (UniqueName: \"kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247420 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f478dce9-8b45-4beb-85ba-0ecc9692cf67-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.247540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.247752 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247569 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247752 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.247752 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247752 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247686 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.247752 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.247741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt4wj\" (UniqueName: \"kubernetes.io/projected/f478dce9-8b45-4beb-85ba-0ecc9692cf67-kube-api-access-xt4wj\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.348500 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt4wj\" (UniqueName: \"kubernetes.io/projected/f478dce9-8b45-4beb-85ba-0ecc9692cf67-kube-api-access-xt4wj\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xj4d\" (UniqueName: \"kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f478dce9-8b45-4beb-85ba-0ecc9692cf67-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.348676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.348985 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.348985 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.348985 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.348766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.349513 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.349416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.349513 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.349427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.349513 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.349485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f478dce9-8b45-4beb-85ba-0ecc9692cf67-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.350039 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.349967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.351963 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.351940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.352064 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.351981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.352307 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.352287 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.352838 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.352817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f478dce9-8b45-4beb-85ba-0ecc9692cf67-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.357515 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.357477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt4wj\" (UniqueName: \"kubernetes.io/projected/f478dce9-8b45-4beb-85ba-0ecc9692cf67-kube-api-access-xt4wj\") pod \"prometheus-operator-68bcd7f987-9lxtn\" (UID: \"f478dce9-8b45-4beb-85ba-0ecc9692cf67\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.358070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.358048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xj4d\" (UniqueName: \"kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d\") pod \"console-7b58ff77cf-mncfs\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.425867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.425798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:46.456191 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.456165 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" Sep 16 18:46:46.716679 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.716600 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:46:46.722967 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:46.722942 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf082145_4213_4183_b233_ba026b6f9f25.slice/crio-c81e39b61a0ec41a26bda4523e623bf8b0c4a04bc48d80e088a3e8a8c64e18e9 WatchSource:0}: Error finding container c81e39b61a0ec41a26bda4523e623bf8b0c4a04bc48d80e088a3e8a8c64e18e9: Status 404 returned error can't find the container with id c81e39b61a0ec41a26bda4523e623bf8b0c4a04bc48d80e088a3e8a8c64e18e9 Sep 16 18:46:46.747675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.747652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn"] Sep 16 18:46:46.749695 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:46.749658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf478dce9_8b45_4beb_85ba_0ecc9692cf67.slice/crio-2c62efd98977cab2f1322a875c86ffe6c667d4c97303fd30bea1cab075ace39c WatchSource:0}: Error finding container 2c62efd98977cab2f1322a875c86ffe6c667d4c97303fd30bea1cab075ace39c: Status 404 returned error can't find the container with id 2c62efd98977cab2f1322a875c86ffe6c667d4c97303fd30bea1cab075ace39c Sep 16 18:46:46.827052 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.827016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" event={"ID":"f478dce9-8b45-4beb-85ba-0ecc9692cf67","Type":"ContainerStarted","Data":"2c62efd98977cab2f1322a875c86ffe6c667d4c97303fd30bea1cab075ace39c"} Sep 16 18:46:46.828535 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:46.828504 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b58ff77cf-mncfs" event={"ID":"bf082145-4213-4183-b233-ba026b6f9f25","Type":"ContainerStarted","Data":"c81e39b61a0ec41a26bda4523e623bf8b0c4a04bc48d80e088a3e8a8c64e18e9"} Sep 16 18:46:48.841824 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:48.841784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" event={"ID":"f478dce9-8b45-4beb-85ba-0ecc9692cf67","Type":"ContainerStarted","Data":"64402a7cb748b400329ace9f976e299fa0dd9b3e0380ee4c767bb95c2de56b6c"} Sep 16 18:46:48.842306 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:48.841894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" event={"ID":"f478dce9-8b45-4beb-85ba-0ecc9692cf67","Type":"ContainerStarted","Data":"e8ffa50809e749b3a8d3641293fe81ad9d57238d3246057a362cd52c494d19e7"} Sep 16 18:46:48.861263 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:48.861205 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9lxtn" podStartSLOduration=1.46895597 podStartE2EDuration="2.861188735s" totalImagesPullingTime="1.392232765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:48.859050381 +0000 UTC m=+138.024070727" watchObservedRunningTime="2026-09-16 18:46:48.861188735 +0000 UTC m=+138.026209074" Sep 16 18:46:50.635573 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.635147 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c9kd2"] Sep 16 18:46:50.639510 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.639486 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.645417 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.645203 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:46:50.645850 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.645821 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:46:50.645850 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.645833 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:46:50.646227 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.646208 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-22rpb\"" Sep 16 18:46:50.682363 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-sys\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682518 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-metrics-client-ca\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682518 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682518 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682690 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-tls\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682690 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-wtmp\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682690 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-root\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682690 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-textfile\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.682885 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.682689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68tb\" (UniqueName: \"kubernetes.io/projected/3b295457-aa8a-4e18-bad5-8572057a242b-kube-api-access-w68tb\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783369 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-root\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-textfile\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-root\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68tb\" (UniqueName: \"kubernetes.io/projected/3b295457-aa8a-4e18-bad5-8572057a242b-kube-api-access-w68tb\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-sys\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-metrics-client-ca\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-tls\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-sys\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-wtmp\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.783867 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-textfile\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.784081 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.783903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-wtmp\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.784241 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.784195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.784393 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.784307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b295457-aa8a-4e18-bad5-8572057a242b-metrics-client-ca\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.786243 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.786222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.786377 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.786359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b295457-aa8a-4e18-bad5-8572057a242b-node-exporter-tls\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.798049 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.798021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68tb\" (UniqueName: \"kubernetes.io/projected/3b295457-aa8a-4e18-bad5-8572057a242b-kube-api-access-w68tb\") pod \"node-exporter-c9kd2\" (UID: \"3b295457-aa8a-4e18-bad5-8572057a242b\") " pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.853380 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.853354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b58ff77cf-mncfs" event={"ID":"bf082145-4213-4183-b233-ba026b6f9f25","Type":"ContainerStarted","Data":"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89"} Sep 16 18:46:50.875946 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.875897 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b58ff77cf-mncfs" podStartSLOduration=1.5902882200000001 podStartE2EDuration="4.875879157s" totalImagesPullingTime="3.285590937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:50.874602997 +0000 UTC m=+140.039623341" watchObservedRunningTime="2026-09-16 18:46:50.875879157 +0000 UTC m=+140.040899500" Sep 16 18:46:50.950212 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:50.950134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9kd2" Sep 16 18:46:50.963254 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:46:50.963220 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b295457_aa8a_4e18_bad5_8572057a242b.slice/crio-84826cd3c39b53a6dbec99977f066b932f30a1a75cae7b05a5cc4cfb38a4ef85 WatchSource:0}: Error finding container 84826cd3c39b53a6dbec99977f066b932f30a1a75cae7b05a5cc4cfb38a4ef85: Status 404 returned error can't find the container with id 84826cd3c39b53a6dbec99977f066b932f30a1a75cae7b05a5cc4cfb38a4ef85 Sep 16 18:46:51.860275 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:51.860064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9kd2" event={"ID":"3b295457-aa8a-4e18-bad5-8572057a242b","Type":"ContainerStarted","Data":"84826cd3c39b53a6dbec99977f066b932f30a1a75cae7b05a5cc4cfb38a4ef85"} Sep 16 18:46:52.866277 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:52.866238 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b295457-aa8a-4e18-bad5-8572057a242b" containerID="2d634df01eb49b76ea7da362652febffa4b9303f55c2014f63024f5303fb651d" exitCode=0 Sep 16 18:46:52.866740 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:52.866298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9kd2" event={"ID":"3b295457-aa8a-4e18-bad5-8572057a242b","Type":"ContainerDied","Data":"2d634df01eb49b76ea7da362652febffa4b9303f55c2014f63024f5303fb651d"} Sep 16 18:46:53.553061 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.552767 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57496669bc-t98jp"] Sep 16 18:46:53.557098 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.557075 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.560463 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 18:46:53.560646 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560414 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 18:46:53.560646 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560431 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5tqsj\"" Sep 16 18:46:53.560646 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 18:46:53.560646 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560580 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 18:46:53.560646 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560531 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e4vjsforom9jd\"" Sep 16 18:46:53.560932 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.560774 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 18:46:53.572887 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.572855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57496669bc-t98jp"] Sep 16 18:46:53.604677 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604801 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604801 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604801 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnq27\" (UniqueName: \"kubernetes.io/projected/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-kube-api-access-dnq27\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604950 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604950 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-metrics-client-ca\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604950 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.604950 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.604933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-grpc-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.705792 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-metrics-client-ca\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.705970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.705970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-grpc-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.705970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.705970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.706168 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.705986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.706168 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.706030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnq27\" (UniqueName: \"kubernetes.io/projected/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-kube-api-access-dnq27\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.706168 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.706059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.706676 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.706555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-metrics-client-ca\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709142 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709245 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709245 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709352 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709393 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.709770 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.709749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-secret-grpc-tls\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.715862 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.715838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnq27\" (UniqueName: \"kubernetes.io/projected/f7573046-c4f6-4bc0-818c-1a38bbfd03a8-kube-api-access-dnq27\") pod \"thanos-querier-57496669bc-t98jp\" (UID: \"f7573046-c4f6-4bc0-818c-1a38bbfd03a8\") " pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.868193 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.868092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:46:53.873077 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.873010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9kd2" event={"ID":"3b295457-aa8a-4e18-bad5-8572057a242b","Type":"ContainerStarted","Data":"8ff5dc400bfb72e546fcc7080af600bec5285ab336a060821d498df4258161e7"} Sep 16 18:46:53.873077 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.873070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9kd2" event={"ID":"3b295457-aa8a-4e18-bad5-8572057a242b","Type":"ContainerStarted","Data":"e83107a7c56d547994fffed9f1eabfd6b9e13313a327ff83fd57140a738645e1"} Sep 16 18:46:53.892469 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:53.892413 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c9kd2" podStartSLOduration=2.789009235 podStartE2EDuration="3.892392287s" totalImagesPullingTime="1.030275925s" totalInitContainerRuntime="73.107127ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.891468461 +0000 UTC m=+143.056488804" watchObservedRunningTime="2026-09-16 18:46:53.892392287 +0000 UTC m=+143.057412629" Sep 16 18:46:55.770004 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.769964 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6459b6f77b-gkmqx"] Sep 16 18:46:55.773823 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.773798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.776841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.776523 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 18:46:55.776841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.776522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 18:46:55.776841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.776633 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 18:46:55.776841 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.776792 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-57jmj\"" Sep 16 18:46:55.777109 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.777001 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 18:46:55.777507 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.777408 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 18:46:55.781985 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.781965 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 18:46:55.787530 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.787504 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6459b6f77b-gkmqx"] Sep 16 18:46:55.822692 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822810 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-serving-certs-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822810 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxxs\" (UniqueName: \"kubernetes.io/projected/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-kube-api-access-flxxs\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822810 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822804 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-metrics-client-ca\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822989 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822989 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822989 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.822989 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.822948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-federate-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924073 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxxs\" (UniqueName: \"kubernetes.io/projected/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-kube-api-access-flxxs\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924252 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-metrics-client-ca\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924252 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924252 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924252 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924252 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-federate-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924571 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924571 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-serving-certs-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.924991 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.924934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-metrics-client-ca\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.925272 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.925236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.925638 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.925592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-serving-certs-ca-bundle\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.927122 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.927095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.927332 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.927299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-telemeter-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.927400 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.927299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-federate-client-tls\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.927751 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.927732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-secret-telemeter-client\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.931434 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.931403 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:46:55.935960 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.935932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:55.936688 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.936668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxxs\" (UniqueName: \"kubernetes.io/projected/6b16c5b7-27e7-43e9-906d-e1e9c5996f26-kube-api-access-flxxs\") pod \"telemeter-client-6459b6f77b-gkmqx\" (UID: \"6b16c5b7-27e7-43e9-906d-e1e9c5996f26\") " pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:55.943144 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.943090 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 18:46:55.946060 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:55.946041 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:46:56.024946 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.024861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.024946 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.024908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsxr\" (UniqueName: \"kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.025161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.024992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.025161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.025012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.025161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.025050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.025161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.025096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.025161 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.025148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.087012 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.086976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" Sep 16 18:46:56.126038 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.125993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126196 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126196 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126196 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126396 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsxr\" (UniqueName: \"kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126396 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126396 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.126833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.126806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.127154 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.127132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.127202 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.127132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.127452 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.127422 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.128929 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.128907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.129140 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.129116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.134649 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.134627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsxr\" (UniqueName: \"kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr\") pod \"console-5c4b598867-pvwsq\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.255747 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.255708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:46:56.426398 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.426360 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:56.426587 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.426435 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:56.431809 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.431785 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:46:56.895440 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:46:56.895411 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:47:01.099462 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.099384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57496669bc-t98jp"] Sep 16 18:47:01.101404 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.101357 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:47:01.103076 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.103049 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6459b6f77b-gkmqx"] Sep 16 18:47:01.105256 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:47:01.105230 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7573046_c4f6_4bc0_818c_1a38bbfd03a8.slice/crio-46f3e600786b4be1038796c2c360986654d829a4feea4943c9740b6f076e61be WatchSource:0}: Error finding container 46f3e600786b4be1038796c2c360986654d829a4feea4943c9740b6f076e61be: Status 404 returned error can't find the container with id 46f3e600786b4be1038796c2c360986654d829a4feea4943c9740b6f076e61be Sep 16 18:47:01.105543 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:47:01.105526 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16200770_5f35_41bf_a4f0_37b719bcccdd.slice/crio-e0096257183aecc169350e1ead4711a0d5141ce6123538ac02f0299c33010c59 WatchSource:0}: Error finding container e0096257183aecc169350e1ead4711a0d5141ce6123538ac02f0299c33010c59: Status 404 returned error can't find the container with id e0096257183aecc169350e1ead4711a0d5141ce6123538ac02f0299c33010c59 Sep 16 18:47:01.107495 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:47:01.106841 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b16c5b7_27e7_43e9_906d_e1e9c5996f26.slice/crio-1a869ae5cf19acabd42192caa59c918b901ca9edd84ddad38a655e4f6e63d2d9 WatchSource:0}: Error finding container 1a869ae5cf19acabd42192caa59c918b901ca9edd84ddad38a655e4f6e63d2d9: Status 404 returned error can't find the container with id 1a869ae5cf19acabd42192caa59c918b901ca9edd84ddad38a655e4f6e63d2d9 Sep 16 18:47:01.918359 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.917619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b598867-pvwsq" event={"ID":"16200770-5f35-41bf-a4f0-37b719bcccdd","Type":"ContainerStarted","Data":"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b"} Sep 16 18:47:01.918359 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.917688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b598867-pvwsq" event={"ID":"16200770-5f35-41bf-a4f0-37b719bcccdd","Type":"ContainerStarted","Data":"e0096257183aecc169350e1ead4711a0d5141ce6123538ac02f0299c33010c59"} Sep 16 18:47:01.924584 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.923756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t6mt7" event={"ID":"362e2e82-5781-437b-bc02-916e42f9e89f","Type":"ContainerStarted","Data":"88894f49a7bcdc6d7d2ca0855aab1499e2431185e3471623093fe4b78e400825"} Sep 16 18:47:01.924584 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.924525 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:47:01.929111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.929010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"46f3e600786b4be1038796c2c360986654d829a4feea4943c9740b6f076e61be"} Sep 16 18:47:01.932148 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.932054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" event={"ID":"6b16c5b7-27e7-43e9-906d-e1e9c5996f26","Type":"ContainerStarted","Data":"1a869ae5cf19acabd42192caa59c918b901ca9edd84ddad38a655e4f6e63d2d9"} Sep 16 18:47:01.937446 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.937396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-t6mt7" Sep 16 18:47:01.945126 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.944910 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c4b598867-pvwsq" podStartSLOduration=6.944892381 podStartE2EDuration="6.944892381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.94407459 +0000 UTC m=+151.109094934" watchObservedRunningTime="2026-09-16 18:47:01.944892381 +0000 UTC m=+151.109912732" Sep 16 18:47:01.975790 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:01.973859 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-t6mt7" podStartSLOduration=2.038269891 podStartE2EDuration="18.973836824s" totalImagesPullingTime="16.935566933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.97345961 +0000 UTC m=+151.138479953" watchObservedRunningTime="2026-09-16 18:47:01.973836824 +0000 UTC m=+151.138857167" Sep 16 18:47:04.952341 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:04.952279 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"92ae773e37f480d995787a4319e727b7ce93d0536da41c6b2c59932cc8ec2cff"} Sep 16 18:47:04.952731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:04.952375 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"b81875899fd980b5dcde1163355f5efb90b39bdb0248056fe9292dcd007e4f0b"} Sep 16 18:47:04.952731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:04.952423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"8960afe368c0b490b5771ca35f5b613a85a4d022420c324e016c13868005c509"} Sep 16 18:47:04.954182 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:04.954152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" event={"ID":"6b16c5b7-27e7-43e9-906d-e1e9c5996f26","Type":"ContainerStarted","Data":"baa99c5241ee843d561330495a8bb3e74180256739caec20d5c2a162770bd264"} Sep 16 18:47:05.828108 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:05.828078 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-959db6b9-vbg5r" Sep 16 18:47:06.256715 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.256630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:06.257237 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.256741 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:06.262216 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.262192 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:06.966890 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.966844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"75a66a8dde10155646f8e5b7de01213f1c98917c3f44b056d63e1fab8872cf26"} Sep 16 18:47:06.967067 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.966933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"e2fe65d8c546d4a8804ed777eac07a4b1ff8d4d1f369d8ba7d13aa885d5ec069"} Sep 16 18:47:06.967067 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.966972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" event={"ID":"f7573046-c4f6-4bc0-818c-1a38bbfd03a8","Type":"ContainerStarted","Data":"4bf9a97aec966d698a2d5a681521a182eaec31f71b31dfa0fdbb4e5180f3b6bb"} Sep 16 18:47:06.969717 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.969682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" event={"ID":"6b16c5b7-27e7-43e9-906d-e1e9c5996f26","Type":"ContainerStarted","Data":"a6ce48ce03b0c99f7f84c6560216cb6425758cd2337c711269e97995c8a30f3a"} Sep 16 18:47:06.969849 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.969741 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" event={"ID":"6b16c5b7-27e7-43e9-906d-e1e9c5996f26","Type":"ContainerStarted","Data":"eb642b14fe09c8837d2073ed85ad0e9987281e729899965ddd88e4383bf6fc74"} Sep 16 18:47:06.974998 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.974972 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:06.992582 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:06.992527 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6459b6f77b-gkmqx" podStartSLOduration=6.725031085 podStartE2EDuration="11.992509758s" totalImagesPullingTime="5.267478673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:06.992080294 +0000 UTC m=+156.157100635" watchObservedRunningTime="2026-09-16 18:47:06.992509758 +0000 UTC m=+156.157530101" Sep 16 18:47:07.055936 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:07.055906 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:47:07.742456 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:47:07.742408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w22ll" podUID="772f55b7-eecf-4c9a-93a6-547a73a0b147" Sep 16 18:47:07.974393 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:07.974360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:08.011229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:08.010160 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" podStartSLOduration=9.981187924 podStartE2EDuration="15.010140182s" totalImagesPullingTime="5.028952258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:08.007566937 +0000 UTC m=+157.172587291" watchObservedRunningTime="2026-09-16 18:47:08.010140182 +0000 UTC m=+157.175160529" Sep 16 18:47:08.965880 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:08.965844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:47:09.002371 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.002343 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:47:09.002541 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.002416 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:47:09.002541 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.002450 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059670 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059670 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059789 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059850 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059800 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.059905 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.059854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrwp\" (UniqueName: \"kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.060080 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.060049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.160970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.160929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161162 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.160990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161162 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161269 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161269 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161408 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161408 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrwp\" (UniqueName: \"kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.161991 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.162099 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.161987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.162099 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.162058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.162219 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.162168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.163913 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.163888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.164021 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.164004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.180423 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.180401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrwp\" (UniqueName: \"kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp\") pod \"console-77fddd6ccd-n2kbr\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.313605 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.313561 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:09.626818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.626757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:47:09.628933 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:47:09.628904 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bce6d0_230c_4391_80f1_f1553236636d.slice/crio-43e00d0652858717e7fd1f45e0e378a41c8a16e48462e9ab30cd44d4d5212ec2 WatchSource:0}: Error finding container 43e00d0652858717e7fd1f45e0e378a41c8a16e48462e9ab30cd44d4d5212ec2: Status 404 returned error can't find the container with id 43e00d0652858717e7fd1f45e0e378a41c8a16e48462e9ab30cd44d4d5212ec2 Sep 16 18:47:09.984836 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.984745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fddd6ccd-n2kbr" event={"ID":"c1bce6d0-230c-4391-80f1-f1553236636d","Type":"ContainerStarted","Data":"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d"} Sep 16 18:47:09.984836 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.984799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fddd6ccd-n2kbr" event={"ID":"c1bce6d0-230c-4391-80f1-f1553236636d","Type":"ContainerStarted","Data":"43e00d0652858717e7fd1f45e0e378a41c8a16e48462e9ab30cd44d4d5212ec2"} Sep 16 18:47:09.989902 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:09.989878 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57496669bc-t98jp" Sep 16 18:47:10.006890 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:10.006846 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77fddd6ccd-n2kbr" podStartSLOduration=2.006833062 podStartE2EDuration="2.006833062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:10.005728832 +0000 UTC m=+159.170749174" watchObservedRunningTime="2026-09-16 18:47:10.006833062 +0000 UTC m=+159.171853403" Sep 16 18:47:12.595856 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:12.595816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:12.598104 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:12.598079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/772f55b7-eecf-4c9a-93a6-547a73a0b147-metrics-tls\") pod \"dns-default-w22ll\" (UID: \"772f55b7-eecf-4c9a-93a6-547a73a0b147\") " pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:12.777871 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:12.777840 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g84ch\"" Sep 16 18:47:12.785971 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:12.785955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:13.086374 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:13.086347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w22ll"] Sep 16 18:47:13.088694 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:47:13.088668 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772f55b7_eecf_4c9a_93a6_547a73a0b147.slice/crio-d3321bc59e8eeeee9dcaf474af1c5845ad98fc9ed714dddd6652a978be447a07 WatchSource:0}: Error finding container d3321bc59e8eeeee9dcaf474af1c5845ad98fc9ed714dddd6652a978be447a07: Status 404 returned error can't find the container with id d3321bc59e8eeeee9dcaf474af1c5845ad98fc9ed714dddd6652a978be447a07 Sep 16 18:47:14.005811 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:14.005772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w22ll" event={"ID":"772f55b7-eecf-4c9a-93a6-547a73a0b147","Type":"ContainerStarted","Data":"d3321bc59e8eeeee9dcaf474af1c5845ad98fc9ed714dddd6652a978be447a07"} Sep 16 18:47:16.015192 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:16.015159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w22ll" event={"ID":"772f55b7-eecf-4c9a-93a6-547a73a0b147","Type":"ContainerStarted","Data":"5d36bdf07a1440b431d4df641d0646c2241591241a2a802df7edc064385d5d5d"} Sep 16 18:47:16.015708 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:16.015240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w22ll" event={"ID":"772f55b7-eecf-4c9a-93a6-547a73a0b147","Type":"ContainerStarted","Data":"4f863c88ab613f21e700df998ac4c253f82a9c12bfcd1959b066d3779044fa44"} Sep 16 18:47:16.015708 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:16.015263 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:16.039374 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:16.039311 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w22ll" podStartSLOduration=130.040669723 podStartE2EDuration="2m12.039270464s" totalImagesPullingTime="1.998600741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:16.038491088 +0000 UTC m=+165.203511431" watchObservedRunningTime="2026-09-16 18:47:16.039270464 +0000 UTC m=+165.204290807" Sep 16 18:47:19.314465 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:19.314425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:19.314852 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:19.314491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:19.319240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:19.319214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:20.034949 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:20.034922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:47:20.091669 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:20.091641 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:47:26.021636 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:26.021563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w22ll" Sep 16 18:47:32.079543 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.079488 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b58ff77cf-mncfs" podUID="bf082145-4213-4183-b233-ba026b6f9f25" containerName="console" containerID="cri-o://728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89" gracePeriod=15 Sep 16 18:47:32.356106 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.356048 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:47:32.455122 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455095 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455271 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455271 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455189 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455387 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455364 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455430 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455396 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xj4d\" (UniqueName: \"kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455430 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455415 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config\") pod \"bf082145-4213-4183-b233-ba026b6f9f25\" (UID: \"bf082145-4213-4183-b233-ba026b6f9f25\") " Sep 16 18:47:32.455588 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455557 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca" (OuterVolumeSpecName: "service-ca") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:32.455694 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455597 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:32.455810 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455794 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-service-ca\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.455853 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455813 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-oauth-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.455887 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.455841 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config" (OuterVolumeSpecName: "console-config") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:32.457538 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.457509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:32.457538 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.457518 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d" (OuterVolumeSpecName: "kube-api-access-9xj4d") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "kube-api-access-9xj4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:47:32.457693 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.457555 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bf082145-4213-4183-b233-ba026b6f9f25" (UID: "bf082145-4213-4183-b233-ba026b6f9f25"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:32.557166 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.557144 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-oauth-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.557166 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.557165 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf082145-4213-4183-b233-ba026b6f9f25-console-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.557279 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.557175 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xj4d\" (UniqueName: \"kubernetes.io/projected/bf082145-4213-4183-b233-ba026b6f9f25-kube-api-access-9xj4d\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.557279 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:32.557186 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf082145-4213-4183-b233-ba026b6f9f25-console-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:33.084847 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.084809 2559 generic.go:356] "Generic (PLEG): container finished" podID="bf082145-4213-4183-b233-ba026b6f9f25" containerID="728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89" exitCode=0 Sep 16 18:47:33.085234 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.084859 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b58ff77cf-mncfs" Sep 16 18:47:33.085234 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.084889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b58ff77cf-mncfs" event={"ID":"bf082145-4213-4183-b233-ba026b6f9f25","Type":"ContainerDied","Data":"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89"} Sep 16 18:47:33.085234 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.084947 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b58ff77cf-mncfs" event={"ID":"bf082145-4213-4183-b233-ba026b6f9f25","Type":"ContainerDied","Data":"c81e39b61a0ec41a26bda4523e623bf8b0c4a04bc48d80e088a3e8a8c64e18e9"} Sep 16 18:47:33.085234 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.084967 2559 scope.go:117] "RemoveContainer" containerID="728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89" Sep 16 18:47:33.093782 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.093764 2559 scope.go:117] "RemoveContainer" containerID="728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89" Sep 16 18:47:33.094047 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:47:33.094028 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89\": container with ID starting with 728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89 not found: ID does not exist" containerID="728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89" Sep 16 18:47:33.094121 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.094052 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89"} err="failed to get container status \"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89\": rpc error: code = NotFound desc = could not find container \"728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89\": container with ID starting with 728f6ea3c67be0dc6af0d2a4c2f816dec0c6b5698836c7503d84c495cbacea89 not found: ID does not exist" Sep 16 18:47:33.107022 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.106999 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:47:33.111697 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.111675 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b58ff77cf-mncfs"] Sep 16 18:47:33.353507 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:33.353436 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf082145-4213-4183-b233-ba026b6f9f25" path="/var/lib/kubelet/pods/bf082145-4213-4183-b233-ba026b6f9f25/volumes" Sep 16 18:47:41.118604 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:41.118517 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae82780d-8d4b-43d4-a9f7-a2926d86ba1b" containerID="18871211d56d4726c9c20636bc18749e85d9a950e9e9029b84fdf2e10e8fd171" exitCode=0 Sep 16 18:47:41.118604 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:41.118594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" event={"ID":"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b","Type":"ContainerDied","Data":"18871211d56d4726c9c20636bc18749e85d9a950e9e9029b84fdf2e10e8fd171"} Sep 16 18:47:41.119033 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:41.118936 2559 scope.go:117] "RemoveContainer" containerID="18871211d56d4726c9c20636bc18749e85d9a950e9e9029b84fdf2e10e8fd171" Sep 16 18:47:42.124112 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:42.124075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lx4jp" event={"ID":"ae82780d-8d4b-43d4-a9f7-a2926d86ba1b","Type":"ContainerStarted","Data":"5bdc8138e8c175744423246d4c992bf9be21ecada50c2927997c5cbc874e5ab6"} Sep 16 18:47:45.110656 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.110611 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c4b598867-pvwsq" podUID="16200770-5f35-41bf-a4f0-37b719bcccdd" containerName="console" containerID="cri-o://f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b" gracePeriod=15 Sep 16 18:47:45.382806 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.382752 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:45.462408 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462384 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462559 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462426 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462559 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462451 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462635 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snsxr\" (UniqueName: \"kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462833 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462785 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle\") pod \"16200770-5f35-41bf-a4f0-37b719bcccdd\" (UID: \"16200770-5f35-41bf-a4f0-37b719bcccdd\") " Sep 16 18:47:45.462932 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.462818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca" (OuterVolumeSpecName: "service-ca") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:45.463050 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.463027 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-service-ca\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.463125 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.463101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:45.463241 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.463216 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:45.463364 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.463271 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config" (OuterVolumeSpecName: "console-config") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:45.464835 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.464806 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr" (OuterVolumeSpecName: "kube-api-access-snsxr") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "kube-api-access-snsxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:47:45.464965 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.464946 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:45.465006 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.464966 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16200770-5f35-41bf-a4f0-37b719bcccdd" (UID: "16200770-5f35-41bf-a4f0-37b719bcccdd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:45.563367 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563344 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.563367 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563365 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-oauth-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.563496 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563375 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsxr\" (UniqueName: \"kubernetes.io/projected/16200770-5f35-41bf-a4f0-37b719bcccdd-kube-api-access-snsxr\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.563496 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563384 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-console-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.563496 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563393 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16200770-5f35-41bf-a4f0-37b719bcccdd-trusted-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:45.563496 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:45.563400 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16200770-5f35-41bf-a4f0-37b719bcccdd-console-oauth-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:47:46.140800 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.140763 2559 generic.go:356] "Generic (PLEG): container finished" podID="16200770-5f35-41bf-a4f0-37b719bcccdd" containerID="f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b" exitCode=0 Sep 16 18:47:46.141286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.140826 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b598867-pvwsq" Sep 16 18:47:46.141286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.140848 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b598867-pvwsq" event={"ID":"16200770-5f35-41bf-a4f0-37b719bcccdd","Type":"ContainerDied","Data":"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b"} Sep 16 18:47:46.141286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.140949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b598867-pvwsq" event={"ID":"16200770-5f35-41bf-a4f0-37b719bcccdd","Type":"ContainerDied","Data":"e0096257183aecc169350e1ead4711a0d5141ce6123538ac02f0299c33010c59"} Sep 16 18:47:46.141286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.140979 2559 scope.go:117] "RemoveContainer" containerID="f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b" Sep 16 18:47:46.149767 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.149751 2559 scope.go:117] "RemoveContainer" containerID="f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b" Sep 16 18:47:46.150041 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:47:46.150019 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b\": container with ID starting with f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b not found: ID does not exist" containerID="f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b" Sep 16 18:47:46.150086 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.150049 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b"} err="failed to get container status \"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b\": rpc error: code = NotFound desc = could not find container \"f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b\": container with ID starting with f0990d355ca22cd67d8d388e58374704dcaea018eb4c0ab2b9f8df5708ef673b not found: ID does not exist" Sep 16 18:47:46.163916 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.163893 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:47:46.168331 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:46.168300 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c4b598867-pvwsq"] Sep 16 18:47:47.353576 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:47:47.353534 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16200770-5f35-41bf-a4f0-37b719bcccdd" path="/var/lib/kubelet/pods/16200770-5f35-41bf-a4f0-37b719bcccdd/volumes" Sep 16 18:48:12.894801 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.894768 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895068 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16200770-5f35-41bf-a4f0-37b719bcccdd" containerName="console" Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895081 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16200770-5f35-41bf-a4f0-37b719bcccdd" containerName="console" Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895095 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf082145-4213-4183-b233-ba026b6f9f25" containerName="console" Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895100 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf082145-4213-4183-b233-ba026b6f9f25" containerName="console" Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895156 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16200770-5f35-41bf-a4f0-37b719bcccdd" containerName="console" Sep 16 18:48:12.895240 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.895164 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf082145-4213-4183-b233-ba026b6f9f25" containerName="console" Sep 16 18:48:12.898815 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.898800 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:12.912877 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912852 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 18:48:12.913070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912853 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 18:48:12.913070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912885 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 18:48:12.913070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912853 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 18:48:12.913070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912892 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 18:48:12.913070 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.912856 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 18:48:12.913328 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.913301 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 18:48:12.913510 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.913493 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zss6w\"" Sep 16 18:48:12.913577 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.913513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 18:48:12.928469 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.928450 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 18:48:12.928780 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:12.928762 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:13.002373 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002316 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002538 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6g4p\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-kube-api-access-c6g4p\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002538 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-tls-assets\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002538 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-config-volume\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002622 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002793 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-config-out\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002793 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002793 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002935 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-web-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.002935 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.003011 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.002954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.103887 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.103855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.103887 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.103890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.103937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-config-out\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.103961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104111 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.103996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104299 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-web-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104583 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104665 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104729 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104729 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6g4p\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-kube-api-access-c6g4p\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104830 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-tls-assets\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104830 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-config-volume\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.104931 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.106013 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.105989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.107230 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.104716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.108229 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.108192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/552cd12c-1573-4e04-81a7-a616831e3286-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.108350 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.108194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.112818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.108792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.112818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.108805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-web-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.112818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.109282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.112818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.109442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.113568 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.113545 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/552cd12c-1573-4e04-81a7-a616831e3286-config-out\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.113767 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.113748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.114611 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.114592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/552cd12c-1573-4e04-81a7-a616831e3286-config-volume\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.114664 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.114604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-tls-assets\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.125657 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.125627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6g4p\" (UniqueName: \"kubernetes.io/projected/552cd12c-1573-4e04-81a7-a616831e3286-kube-api-access-c6g4p\") pod \"alertmanager-main-0\" (UID: \"552cd12c-1573-4e04-81a7-a616831e3286\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.207899 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.207831 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:13.499413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:13.499388 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:13.501747 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:48:13.501727 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552cd12c_1573_4e04_81a7_a616831e3286.slice/crio-a0b154b62d2b149cb3eb1c7392756b31aa5f9fe69d1844a6e6aff4a7fa57c937 WatchSource:0}: Error finding container a0b154b62d2b149cb3eb1c7392756b31aa5f9fe69d1844a6e6aff4a7fa57c937: Status 404 returned error can't find the container with id a0b154b62d2b149cb3eb1c7392756b31aa5f9fe69d1844a6e6aff4a7fa57c937 Sep 16 18:48:14.253104 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:14.253063 2559 generic.go:356] "Generic (PLEG): container finished" podID="552cd12c-1573-4e04-81a7-a616831e3286" containerID="44e3194c93db492ee0417cfdc5d45a260532600a5eeda4cc50f7da3bbb6df2d1" exitCode=0 Sep 16 18:48:14.253608 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:14.253128 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerDied","Data":"44e3194c93db492ee0417cfdc5d45a260532600a5eeda4cc50f7da3bbb6df2d1"} Sep 16 18:48:14.253608 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:14.253183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"a0b154b62d2b149cb3eb1c7392756b31aa5f9fe69d1844a6e6aff4a7fa57c937"} Sep 16 18:48:16.264685 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"90650fbb93122dc2421fc104ada7ec5136ba18e303f8fa99ec52c9961b97bdc1"} Sep 16 18:48:16.265127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"23849b7a827e7632bd41f5445cbd0a2c37b850f1372b56af3ae74918cb182315"} Sep 16 18:48:16.265127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264765 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"387f4fe8d510cce0b9a5872b34bc6adeac188053b789d9cfe8d5b08e6e4950a9"} Sep 16 18:48:16.265127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"d5be15b3fcd65548f603c78888ec906c6e0aff6e4d9bf227dccb427b0f2c6d31"} Sep 16 18:48:16.265127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264848 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"ba6c62975be5432cc9309fd51b84f32b1499f038d996bd1a5daef8b670b29462"} Sep 16 18:48:16.265127 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.264884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"552cd12c-1573-4e04-81a7-a616831e3286","Type":"ContainerStarted","Data":"520ac32036e5bf8df4f15ee38dcee611e11397fc26a1ee7bdf92f4bb670e4a5f"} Sep 16 18:48:16.324491 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:16.320271 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.712387567 podStartE2EDuration="4.32024939s" totalImagesPullingTime="1.595044533s" totalInitContainerRuntime="12.81729ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:16.319097742 +0000 UTC m=+225.484118085" watchObservedRunningTime="2026-09-16 18:48:16.32024939 +0000 UTC m=+225.485269732" Sep 16 18:48:35.050808 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.050775 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:48:35.055419 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.055393 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070466 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070605 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlkq\" (UniqueName: \"kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070605 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070569 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070701 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070701 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070701 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.070701 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.070690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.081302 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.081278 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:48:35.171156 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171156 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171425 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171425 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171425 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlkq\" (UniqueName: \"kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.171578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.171479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.172153 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.172116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.172153 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.172133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.172354 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.172132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.172476 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.172458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.173699 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.173679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.173847 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.173829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.180332 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.180298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlkq\" (UniqueName: \"kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq\") pod \"console-64b796cfb-kchz9\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.366265 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.366188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:35.644197 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:35.644121 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:48:35.647500 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:48:35.647460 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd014e600_0f92_42a5_86a3_9e20d69cd223.slice/crio-90f0dcdf7a0bf92563e139c2cab348ff09975712af4fc04728d82d92b933780a WatchSource:0}: Error finding container 90f0dcdf7a0bf92563e139c2cab348ff09975712af4fc04728d82d92b933780a: Status 404 returned error can't find the container with id 90f0dcdf7a0bf92563e139c2cab348ff09975712af4fc04728d82d92b933780a Sep 16 18:48:36.346006 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:36.345972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b796cfb-kchz9" event={"ID":"d014e600-0f92-42a5-86a3-9e20d69cd223","Type":"ContainerStarted","Data":"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953"} Sep 16 18:48:36.346459 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:36.346023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b796cfb-kchz9" event={"ID":"d014e600-0f92-42a5-86a3-9e20d69cd223","Type":"ContainerStarted","Data":"90f0dcdf7a0bf92563e139c2cab348ff09975712af4fc04728d82d92b933780a"} Sep 16 18:48:36.368231 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:36.368183 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b796cfb-kchz9" podStartSLOduration=1.36816822 podStartE2EDuration="1.36816822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.366351617 +0000 UTC m=+245.531371959" watchObservedRunningTime="2026-09-16 18:48:36.36816822 +0000 UTC m=+245.533188562" Sep 16 18:48:45.366975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:45.366941 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:45.367522 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:45.367000 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:45.371832 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:45.371810 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:45.384404 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:45.384385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:48:45.500131 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:48:45.500091 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:49:10.519155 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.519119 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77fddd6ccd-n2kbr" podUID="c1bce6d0-230c-4391-80f1-f1553236636d" containerName="console" containerID="cri-o://cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d" gracePeriod=15 Sep 16 18:49:10.758023 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.757998 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:49:10.865376 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865350 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865376 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865383 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865590 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865429 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865590 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865478 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865590 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865514 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865590 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmrwp\" (UniqueName: \"kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865826 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865753 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle\") pod \"c1bce6d0-230c-4391-80f1-f1553236636d\" (UID: \"c1bce6d0-230c-4391-80f1-f1553236636d\") " Sep 16 18:49:10.865826 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865784 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca" (OuterVolumeSpecName: "service-ca") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.866055 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.865945 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.866137 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.866103 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-service-ca\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.866137 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.866123 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-oauth-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.866217 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.866129 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.866217 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.866159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config" (OuterVolumeSpecName: "console-config") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:10.867716 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.867697 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:10.868029 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.868003 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:10.868029 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.868010 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp" (OuterVolumeSpecName: "kube-api-access-vmrwp") pod "c1bce6d0-230c-4391-80f1-f1553236636d" (UID: "c1bce6d0-230c-4391-80f1-f1553236636d"). InnerVolumeSpecName "kube-api-access-vmrwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:10.967437 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.967402 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrwp\" (UniqueName: \"kubernetes.io/projected/c1bce6d0-230c-4391-80f1-f1553236636d-kube-api-access-vmrwp\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.967437 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.967432 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-trusted-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.967437 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.967441 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.967674 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.967451 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1bce6d0-230c-4391-80f1-f1553236636d-console-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:10.967674 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:10.967461 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1bce6d0-230c-4391-80f1-f1553236636d-console-oauth-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:11.482999 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.482966 2559 generic.go:356] "Generic (PLEG): container finished" podID="c1bce6d0-230c-4391-80f1-f1553236636d" containerID="cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d" exitCode=0 Sep 16 18:49:11.483156 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.483021 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fddd6ccd-n2kbr" Sep 16 18:49:11.483156 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.483023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fddd6ccd-n2kbr" event={"ID":"c1bce6d0-230c-4391-80f1-f1553236636d","Type":"ContainerDied","Data":"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d"} Sep 16 18:49:11.483236 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.483167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fddd6ccd-n2kbr" event={"ID":"c1bce6d0-230c-4391-80f1-f1553236636d","Type":"ContainerDied","Data":"43e00d0652858717e7fd1f45e0e378a41c8a16e48462e9ab30cd44d4d5212ec2"} Sep 16 18:49:11.483236 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.483199 2559 scope.go:117] "RemoveContainer" containerID="cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d" Sep 16 18:49:11.491432 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.491413 2559 scope.go:117] "RemoveContainer" containerID="cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d" Sep 16 18:49:11.491681 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:11.491665 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d\": container with ID starting with cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d not found: ID does not exist" containerID="cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d" Sep 16 18:49:11.491736 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.491688 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d"} err="failed to get container status \"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d\": rpc error: code = NotFound desc = could not find container \"cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d\": container with ID starting with cc40caeed69ab3251a5bc6ce180db98e2c0f5bcb102413d950e16fa1bc396d0d not found: ID does not exist" Sep 16 18:49:11.501385 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.501364 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:49:11.508149 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:11.508128 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77fddd6ccd-n2kbr"] Sep 16 18:49:13.353063 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:13.353029 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1bce6d0-230c-4391-80f1-f1553236636d" path="/var/lib/kubelet/pods/c1bce6d0-230c-4391-80f1-f1553236636d/volumes" Sep 16 18:49:31.258081 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.258052 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:49:31.264006 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.263981 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:49:31.271100 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.270633 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:49:31.273339 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.273296 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s"] Sep 16 18:49:31.273789 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.273768 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1bce6d0-230c-4391-80f1-f1553236636d" containerName="console" Sep 16 18:49:31.273789 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.273790 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1bce6d0-230c-4391-80f1-f1553236636d" containerName="console" Sep 16 18:49:31.273942 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.273902 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1bce6d0-230c-4391-80f1-f1553236636d" containerName="console" Sep 16 18:49:31.277398 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.277381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.281722 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.281707 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 18:49:31.283872 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.283852 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 18:49:31.283962 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.283883 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4f7j4\"" Sep 16 18:49:31.289187 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.289170 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s"] Sep 16 18:49:31.430779 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.430641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.430779 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.430668 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.430779 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.430720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtmb\" (UniqueName: \"kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.531796 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.531724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtmb\" (UniqueName: \"kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.531796 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.531788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.531969 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.531807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.532205 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.532186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.532250 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.532191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.539921 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.539889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtmb\" (UniqueName: \"kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.586112 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.586085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:31.857126 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:31.857101 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s"] Sep 16 18:49:31.859092 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:49:31.859055 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd27889_cf11_480d_a109_28d995ea0c17.slice/crio-62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f WatchSource:0}: Error finding container 62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f: Status 404 returned error can't find the container with id 62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f Sep 16 18:49:32.569811 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:32.569760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" event={"ID":"7cd27889-cf11-480d-a109-28d995ea0c17","Type":"ContainerStarted","Data":"62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f"} Sep 16 18:49:38.598576 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:38.598538 2559 generic.go:356] "Generic (PLEG): container finished" podID="7cd27889-cf11-480d-a109-28d995ea0c17" containerID="197a450c0263a8147a8ca2d3f80d79362f31d81da03b2d5d9d2aff71ff0a3907" exitCode=0 Sep 16 18:49:38.599033 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:38.598619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" event={"ID":"7cd27889-cf11-480d-a109-28d995ea0c17","Type":"ContainerDied","Data":"197a450c0263a8147a8ca2d3f80d79362f31d81da03b2d5d9d2aff71ff0a3907"} Sep 16 18:49:38.599700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:38.599677 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:49:40.609160 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:40.609126 2559 generic.go:356] "Generic (PLEG): container finished" podID="7cd27889-cf11-480d-a109-28d995ea0c17" containerID="a24e29024fe197e461878f39f3628cf1450a6b44b32666e2bf2671cdb77570d7" exitCode=0 Sep 16 18:49:40.609504 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:40.609192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" event={"ID":"7cd27889-cf11-480d-a109-28d995ea0c17","Type":"ContainerDied","Data":"a24e29024fe197e461878f39f3628cf1450a6b44b32666e2bf2671cdb77570d7"} Sep 16 18:49:48.644469 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:48.644429 2559 generic.go:356] "Generic (PLEG): container finished" podID="7cd27889-cf11-480d-a109-28d995ea0c17" containerID="dbb5985b29c592644a6b9799e01efa8de73d9e1d50731f416f0fe32c5063a51e" exitCode=0 Sep 16 18:49:48.644902 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:48.644508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" event={"ID":"7cd27889-cf11-480d-a109-28d995ea0c17","Type":"ContainerDied","Data":"dbb5985b29c592644a6b9799e01efa8de73d9e1d50731f416f0fe32c5063a51e"} Sep 16 18:49:49.776822 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.776798 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:49.889992 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.889964 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util\") pod \"7cd27889-cf11-480d-a109-28d995ea0c17\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " Sep 16 18:49:49.890137 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.890019 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgtmb\" (UniqueName: \"kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb\") pod \"7cd27889-cf11-480d-a109-28d995ea0c17\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " Sep 16 18:49:49.890137 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.890078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle\") pod \"7cd27889-cf11-480d-a109-28d995ea0c17\" (UID: \"7cd27889-cf11-480d-a109-28d995ea0c17\") " Sep 16 18:49:49.890726 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.890702 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle" (OuterVolumeSpecName: "bundle") pod "7cd27889-cf11-480d-a109-28d995ea0c17" (UID: "7cd27889-cf11-480d-a109-28d995ea0c17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:49.892168 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.892148 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb" (OuterVolumeSpecName: "kube-api-access-fgtmb") pod "7cd27889-cf11-480d-a109-28d995ea0c17" (UID: "7cd27889-cf11-480d-a109-28d995ea0c17"). InnerVolumeSpecName "kube-api-access-fgtmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:49.894472 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.894448 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util" (OuterVolumeSpecName: "util") pod "7cd27889-cf11-480d-a109-28d995ea0c17" (UID: "7cd27889-cf11-480d-a109-28d995ea0c17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:49.991343 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.991271 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-util\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:49.991343 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.991295 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtmb\" (UniqueName: \"kubernetes.io/projected/7cd27889-cf11-480d-a109-28d995ea0c17-kube-api-access-fgtmb\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:49.991343 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:49.991306 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cd27889-cf11-480d-a109-28d995ea0c17-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:49:50.653918 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:50.653883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" event={"ID":"7cd27889-cf11-480d-a109-28d995ea0c17","Type":"ContainerDied","Data":"62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f"} Sep 16 18:49:50.653918 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:50.653909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahwx7s" Sep 16 18:49:50.653918 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:50.653926 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c7a6732af97276d6b89cee86a6f2875e3f5f4d9d0db126c229c4193cb2979f" Sep 16 18:49:55.111419 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111386 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78"] Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111720 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="util" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111730 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="util" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111745 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="pull" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111750 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="pull" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111765 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="extract" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111770 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="extract" Sep 16 18:49:55.111861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.111822 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd27889-cf11-480d-a109-28d995ea0c17" containerName="extract" Sep 16 18:49:55.125686 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.125668 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.128268 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.128244 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 18:49:55.128495 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.128480 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 18:49:55.128599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.128580 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tvbls\"" Sep 16 18:49:55.128665 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.128654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 18:49:55.131949 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.131928 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78"] Sep 16 18:49:55.234064 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.234034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.234206 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.234072 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp928\" (UniqueName: \"kubernetes.io/projected/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-kube-api-access-mp928\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.334885 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.334855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.335006 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.334900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp928\" (UniqueName: \"kubernetes.io/projected/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-kube-api-access-mp928\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.337146 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.337121 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.361307 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.361287 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp928\" (UniqueName: \"kubernetes.io/projected/dbb7c8bc-9a40-48b4-ac3d-16de9631f332-kube-api-access-mp928\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-qnr78\" (UID: \"dbb7c8bc-9a40-48b4-ac3d-16de9631f332\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.435974 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.435910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:55.724753 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:55.724673 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78"] Sep 16 18:49:55.727846 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:49:55.727814 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb7c8bc_9a40_48b4_ac3d_16de9631f332.slice/crio-040402dfeb0a33d96882333cb048c1f52616da10d7a86be4752fd8301eac2010 WatchSource:0}: Error finding container 040402dfeb0a33d96882333cb048c1f52616da10d7a86be4752fd8301eac2010: Status 404 returned error can't find the container with id 040402dfeb0a33d96882333cb048c1f52616da10d7a86be4752fd8301eac2010 Sep 16 18:49:56.681749 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:56.681712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" event={"ID":"dbb7c8bc-9a40-48b4-ac3d-16de9631f332","Type":"ContainerStarted","Data":"040402dfeb0a33d96882333cb048c1f52616da10d7a86be4752fd8301eac2010"} Sep 16 18:49:58.693539 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:58.693433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" event={"ID":"dbb7c8bc-9a40-48b4-ac3d-16de9631f332","Type":"ContainerStarted","Data":"bb7ee7e6250f07a917449b156c1fd5d12b33d625ca960190c1dc8f66b438ae44"} Sep 16 18:49:58.693908 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:58.693573 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:49:58.714239 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:58.714193 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" podStartSLOduration=1.005474368 podStartE2EDuration="3.714178128s" totalImagesPullingTime="2.70870376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:58.712606892 +0000 UTC m=+327.877627247" watchObservedRunningTime="2026-09-16 18:49:58.714178128 +0000 UTC m=+327.879198471" Sep 16 18:49:59.176121 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.176089 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-g8ggb"] Sep 16 18:49:59.179545 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.179525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.182289 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.182267 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 16 18:49:59.182428 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.182361 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 18:49:59.182428 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.182390 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-d85hl\"" Sep 16 18:49:59.195186 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.195165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-g8ggb"] Sep 16 18:49:59.267054 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.267020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d210f60-43e6-4bc2-9b1e-9b651d398877-cabundle0\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.267175 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.267099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.267175 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.267153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9qj\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-kube-api-access-qg9qj\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.367928 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.367897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.368073 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.367945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9qj\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-kube-api-access-qg9qj\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.368073 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.367976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d210f60-43e6-4bc2-9b1e-9b651d398877-cabundle0\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.368073 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:59.368054 2559 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Sep 16 18:49:59.368190 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:59.368077 2559 secret.go:281] references non-existent secret key: ca.crt Sep 16 18:49:59.368190 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:59.368089 2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 18:49:59.368190 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:59.368105 2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-g8ggb: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 18:49:59.368190 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:49:59.368160 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates podName:4d210f60-43e6-4bc2-9b1e-9b651d398877 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:59.868141086 +0000 UTC m=+329.033161410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates") pod "keda-operator-579c8958f7-g8ggb" (UID: "4d210f60-43e6-4bc2-9b1e-9b651d398877") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 18:49:59.368584 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.368567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d210f60-43e6-4bc2-9b1e-9b651d398877-cabundle0\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.388928 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.388894 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9qj\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-kube-api-access-qg9qj\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.872003 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.871969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:49:59.874296 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:49:59.874275 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d210f60-43e6-4bc2-9b1e-9b651d398877-certificates\") pod \"keda-operator-579c8958f7-g8ggb\" (UID: \"4d210f60-43e6-4bc2-9b1e-9b651d398877\") " pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:50:00.089457 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:00.089421 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:50:00.365237 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:00.365188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-g8ggb"] Sep 16 18:50:00.367803 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:50:00.367777 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d210f60_43e6_4bc2_9b1e_9b651d398877.slice/crio-4b84da3ffbd9d46ad4225ee255a0aa07682a8ee858bff2b956d00b9ff0167e2e WatchSource:0}: Error finding container 4b84da3ffbd9d46ad4225ee255a0aa07682a8ee858bff2b956d00b9ff0167e2e: Status 404 returned error can't find the container with id 4b84da3ffbd9d46ad4225ee255a0aa07682a8ee858bff2b956d00b9ff0167e2e Sep 16 18:50:00.704561 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:00.704523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" event={"ID":"4d210f60-43e6-4bc2-9b1e-9b651d398877","Type":"ContainerStarted","Data":"4b84da3ffbd9d46ad4225ee255a0aa07682a8ee858bff2b956d00b9ff0167e2e"} Sep 16 18:50:15.773173 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:15.773137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" event={"ID":"4d210f60-43e6-4bc2-9b1e-9b651d398877","Type":"ContainerStarted","Data":"8d167bd64249dd85acbad2c4e6b5b4afc939f9c2a035bb46d9433dcce5408bfd"} Sep 16 18:50:15.773578 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:15.773249 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:50:15.795179 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:15.795134 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" podStartSLOduration=2.046632756 podStartE2EDuration="16.795121333s" totalImagesPullingTime="14.748488577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:15.794375931 +0000 UTC m=+344.959396275" watchObservedRunningTime="2026-09-16 18:50:15.795121333 +0000 UTC m=+344.960141675" Sep 16 18:50:19.699998 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:19.699971 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-qnr78" Sep 16 18:50:36.780207 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:50:36.780124 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-g8ggb" Sep 16 18:51:19.452582 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.452550 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p"] Sep 16 18:51:19.455845 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.455828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:19.458088 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.458066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:51:19.458328 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.458300 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 16 18:51:19.458389 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.458356 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:51:19.459227 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.459205 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ztq7g\"" Sep 16 18:51:19.467439 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.467364 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p"] Sep 16 18:51:19.641808 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.641777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/60686baf-570b-416f-85ea-7e620b9914d8-kube-api-access-496wj\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:19.641967 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.641837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:19.742911 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.742844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:19.742911 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.742907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/60686baf-570b-416f-85ea-7e620b9914d8-kube-api-access-496wj\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:19.743058 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:51:19.742989 2559 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 18:51:19.743058 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:51:19.743055 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert podName:60686baf-570b-416f-85ea-7e620b9914d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:20.243036181 +0000 UTC m=+409.408056518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert") pod "llmisvc-controller-manager-645f9c8fd6-jmz9p" (UID: "60686baf-570b-416f-85ea-7e620b9914d8") : secret "llmisvc-webhook-server-cert" not found Sep 16 18:51:19.752314 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:19.752282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-496wj\" (UniqueName: \"kubernetes.io/projected/60686baf-570b-416f-85ea-7e620b9914d8-kube-api-access-496wj\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:20.247876 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:20.247844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:20.250176 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:20.250147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60686baf-570b-416f-85ea-7e620b9914d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jmz9p\" (UID: \"60686baf-570b-416f-85ea-7e620b9914d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:20.366088 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:20.366053 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:20.653206 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:20.653055 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p"] Sep 16 18:51:20.657365 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:51:20.657336 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60686baf_570b_416f_85ea_7e620b9914d8.slice/crio-d840de565468a78dd7e0c9aae517dbee7af7c8bfc6926e2d30426d2ee2ac992d WatchSource:0}: Error finding container d840de565468a78dd7e0c9aae517dbee7af7c8bfc6926e2d30426d2ee2ac992d: Status 404 returned error can't find the container with id d840de565468a78dd7e0c9aae517dbee7af7c8bfc6926e2d30426d2ee2ac992d Sep 16 18:51:21.058939 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:21.058905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" event={"ID":"60686baf-570b-416f-85ea-7e620b9914d8","Type":"ContainerStarted","Data":"d840de565468a78dd7e0c9aae517dbee7af7c8bfc6926e2d30426d2ee2ac992d"} Sep 16 18:51:25.081336 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:25.081281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" event={"ID":"60686baf-570b-416f-85ea-7e620b9914d8","Type":"ContainerStarted","Data":"8347a2ab73191e55afbb802d84421551fa050a91308935b549bfab7da2cbd56d"} Sep 16 18:51:25.081721 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:25.081453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:51:25.100064 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:25.100016 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" podStartSLOduration=2.51103708 podStartE2EDuration="6.100002237s" totalImagesPullingTime="3.588965157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:25.09846531 +0000 UTC m=+414.263485652" watchObservedRunningTime="2026-09-16 18:51:25.100002237 +0000 UTC m=+414.265022613" Sep 16 18:51:56.088807 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:51:56.088728 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jmz9p" Sep 16 18:52:03.395787 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.395751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8ff9558bd-s2ffn"] Sep 16 18:52:03.401480 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.401459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506364 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-console-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-service-ca\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj64z\" (UniqueName: \"kubernetes.io/projected/3f01324e-4b51-43af-a1d3-96754e46c44d-kube-api-access-tj64z\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-oauth-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-trusted-ca-bundle\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506509 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.506528 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.506523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-oauth-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608105 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-console-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608105 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-service-ca\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608381 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj64z\" (UniqueName: \"kubernetes.io/projected/3f01324e-4b51-43af-a1d3-96754e46c44d-kube-api-access-tj64z\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608381 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-oauth-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608381 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-trusted-ca-bundle\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608381 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608381 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-oauth-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.608970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-service-ca\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.609089 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-oauth-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.609089 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.608939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-console-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.609208 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.609152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f01324e-4b51-43af-a1d3-96754e46c44d-trusted-ca-bundle\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.610771 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.610750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-oauth-config\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.610882 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.610865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f01324e-4b51-43af-a1d3-96754e46c44d-console-serving-cert\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.616768 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.616745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ff9558bd-s2ffn"] Sep 16 18:52:03.620263 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.620239 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj64z\" (UniqueName: \"kubernetes.io/projected/3f01324e-4b51-43af-a1d3-96754e46c44d-kube-api-access-tj64z\") pod \"console-8ff9558bd-s2ffn\" (UID: \"3f01324e-4b51-43af-a1d3-96754e46c44d\") " pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.710958 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.710878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:03.989886 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:03.989862 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ff9558bd-s2ffn"] Sep 16 18:52:03.991686 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:52:03.991659 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f01324e_4b51_43af_a1d3_96754e46c44d.slice/crio-c647be17bed0567c275dcb164927a5f176f29604821078f5e0724e8d71de2bb0 WatchSource:0}: Error finding container c647be17bed0567c275dcb164927a5f176f29604821078f5e0724e8d71de2bb0: Status 404 returned error can't find the container with id c647be17bed0567c275dcb164927a5f176f29604821078f5e0724e8d71de2bb0 Sep 16 18:52:04.260039 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:04.259950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ff9558bd-s2ffn" event={"ID":"3f01324e-4b51-43af-a1d3-96754e46c44d","Type":"ContainerStarted","Data":"87fdccb18027ad83da102685bd44875be18519bce64b09726cf9101dba26fbae"} Sep 16 18:52:04.260039 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:04.259998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ff9558bd-s2ffn" event={"ID":"3f01324e-4b51-43af-a1d3-96754e46c44d","Type":"ContainerStarted","Data":"c647be17bed0567c275dcb164927a5f176f29604821078f5e0724e8d71de2bb0"} Sep 16 18:52:04.284589 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:04.284533 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8ff9558bd-s2ffn" podStartSLOduration=1.284513646 podStartE2EDuration="1.284513646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:04.283494168 +0000 UTC m=+453.448514523" watchObservedRunningTime="2026-09-16 18:52:04.284513646 +0000 UTC m=+453.449533999" Sep 16 18:52:13.711243 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:13.711203 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:13.711704 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:13.711295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:13.716201 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:13.716172 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:14.316668 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:14.316638 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8ff9558bd-s2ffn" Sep 16 18:52:14.380502 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:14.380463 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:52:39.402731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.402673 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64b796cfb-kchz9" podUID="d014e600-0f92-42a5-86a3-9e20d69cd223" containerName="console" containerID="cri-o://5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953" gracePeriod=15 Sep 16 18:52:39.650811 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.650781 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:52:39.691007 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.690941 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691007 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.690991 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691033 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691065 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjlkq\" (UniqueName: \"kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691156 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691198 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691195 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle\") pod \"d014e600-0f92-42a5-86a3-9e20d69cd223\" (UID: \"d014e600-0f92-42a5-86a3-9e20d69cd223\") " Sep 16 18:52:39.691479 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca" (OuterVolumeSpecName: "service-ca") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:52:39.691631 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.691495 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config" (OuterVolumeSpecName: "console-config") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:52:39.694026 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.692061 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-service-ca\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.694026 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.692163 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:52:39.694026 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.692682 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:52:39.695557 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.695527 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:52:39.701409 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.701366 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:52:39.701553 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.701524 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq" (OuterVolumeSpecName: "kube-api-access-tjlkq") pod "d014e600-0f92-42a5-86a3-9e20d69cd223" (UID: "d014e600-0f92-42a5-86a3-9e20d69cd223"). InnerVolumeSpecName "kube-api-access-tjlkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:52:39.753731 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.753696 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dxlzc"] Sep 16 18:52:39.754245 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.754229 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d014e600-0f92-42a5-86a3-9e20d69cd223" containerName="console" Sep 16 18:52:39.754315 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.754248 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d014e600-0f92-42a5-86a3-9e20d69cd223" containerName="console" Sep 16 18:52:39.754413 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.754401 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d014e600-0f92-42a5-86a3-9e20d69cd223" containerName="console" Sep 16 18:52:39.757912 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.757890 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.760311 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.760289 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 18:52:39.760507 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.760488 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-f7cx2\"" Sep 16 18:52:39.767368 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.767347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dxlzc"] Sep 16 18:52:39.792525 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mj4q\" (UniqueName: \"kubernetes.io/projected/657e282d-6828-409e-b268-a29e4815f51d-kube-api-access-9mj4q\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.792700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e282d-6828-409e-b268-a29e4815f51d-tls-certs\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.792700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792665 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-oauth-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.792700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792686 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-trusted-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.792700 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792696 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-oauth-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.792844 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792705 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d014e600-0f92-42a5-86a3-9e20d69cd223-console-config\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.792844 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792713 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjlkq\" (UniqueName: \"kubernetes.io/projected/d014e600-0f92-42a5-86a3-9e20d69cd223-kube-api-access-tjlkq\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.792844 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.792721 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d014e600-0f92-42a5-86a3-9e20d69cd223-console-serving-cert\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:52:39.893503 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.893466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mj4q\" (UniqueName: \"kubernetes.io/projected/657e282d-6828-409e-b268-a29e4815f51d-kube-api-access-9mj4q\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.893668 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.893528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e282d-6828-409e-b268-a29e4815f51d-tls-certs\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.895918 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.895893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e282d-6828-409e-b268-a29e4815f51d-tls-certs\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:39.902369 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:39.902350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mj4q\" (UniqueName: \"kubernetes.io/projected/657e282d-6828-409e-b268-a29e4815f51d-kube-api-access-9mj4q\") pod \"model-serving-api-7b954ffc95-dxlzc\" (UID: \"657e282d-6828-409e-b268-a29e4815f51d\") " pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:40.069652 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.069622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:40.345121 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.345042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dxlzc"] Sep 16 18:52:40.348283 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:52:40.348256 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657e282d_6828_409e_b268_a29e4815f51d.slice/crio-c09cbe9b8e824747fbf7329dafb9a51f19e72516bee40ca9eba1d62a97a2501a WatchSource:0}: Error finding container c09cbe9b8e824747fbf7329dafb9a51f19e72516bee40ca9eba1d62a97a2501a: Status 404 returned error can't find the container with id c09cbe9b8e824747fbf7329dafb9a51f19e72516bee40ca9eba1d62a97a2501a Sep 16 18:52:40.435390 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.435355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dxlzc" event={"ID":"657e282d-6828-409e-b268-a29e4815f51d","Type":"ContainerStarted","Data":"c09cbe9b8e824747fbf7329dafb9a51f19e72516bee40ca9eba1d62a97a2501a"} Sep 16 18:52:40.436751 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.436726 2559 generic.go:356] "Generic (PLEG): container finished" podID="d014e600-0f92-42a5-86a3-9e20d69cd223" containerID="5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953" exitCode=0 Sep 16 18:52:40.436860 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.436778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b796cfb-kchz9" event={"ID":"d014e600-0f92-42a5-86a3-9e20d69cd223","Type":"ContainerDied","Data":"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953"} Sep 16 18:52:40.436860 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.436789 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b796cfb-kchz9" Sep 16 18:52:40.436860 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.436810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b796cfb-kchz9" event={"ID":"d014e600-0f92-42a5-86a3-9e20d69cd223","Type":"ContainerDied","Data":"90f0dcdf7a0bf92563e139c2cab348ff09975712af4fc04728d82d92b933780a"} Sep 16 18:52:40.436860 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.436830 2559 scope.go:117] "RemoveContainer" containerID="5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953" Sep 16 18:52:40.445367 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.445347 2559 scope.go:117] "RemoveContainer" containerID="5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953" Sep 16 18:52:40.445619 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:52:40.445601 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953\": container with ID starting with 5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953 not found: ID does not exist" containerID="5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953" Sep 16 18:52:40.445687 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.445631 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953"} err="failed to get container status \"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953\": rpc error: code = NotFound desc = could not find container \"5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953\": container with ID starting with 5131758fd1605d82ff7c43230653bbdbeb4c1f8140867ae21c45bbac9bd77953 not found: ID does not exist" Sep 16 18:52:40.464922 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.464889 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:52:40.471911 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:40.471890 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64b796cfb-kchz9"] Sep 16 18:52:41.356385 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:41.356354 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d014e600-0f92-42a5-86a3-9e20d69cd223" path="/var/lib/kubelet/pods/d014e600-0f92-42a5-86a3-9e20d69cd223/volumes" Sep 16 18:52:43.455635 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:43.455597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dxlzc" event={"ID":"657e282d-6828-409e-b268-a29e4815f51d","Type":"ContainerStarted","Data":"26b7600c44347edacb0ca8f786938e0d7e2c0bd9d83820f6ac51bb7dfc2eabc1"} Sep 16 18:52:43.456074 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:43.455708 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:54.464234 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:54.464195 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dxlzc" Sep 16 18:52:54.481659 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:52:54.481609 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dxlzc" podStartSLOduration=13.139616075 podStartE2EDuration="15.481594053s" totalImagesPullingTime="2.341977978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:43.492038327 +0000 UTC m=+492.657058669" watchObservedRunningTime="2026-09-16 18:52:54.481594053 +0000 UTC m=+503.646614394" Sep 16 18:54:31.291011 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:54:31.290985 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:54:31.296671 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:54:31.296640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:55:55.625246 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.625211 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:55:55.628556 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.628540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.632682 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.632661 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:55:55.632807 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.632661 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-aa795-serving-cert\"" Sep 16 18:55:55.632807 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.632667 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hvpnn\"" Sep 16 18:55:55.632807 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.632667 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-aa795-kube-rbac-proxy-sar-config\"" Sep 16 18:55:55.684128 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.684098 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:55:55.717335 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.717283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.717469 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.717398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.818807 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.818779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.818975 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.818850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.819422 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.819395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.821177 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.821157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls\") pod \"switch-graph-aa795-cf8569c7b-m8kxr\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:55.938455 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:55.938389 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:56.210622 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:56.210567 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:55:56.212992 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:55:56.212960 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767daf61_7d49_4541_9057_b60f19d2d7e4.slice/crio-1030fc50781f10a23d9ecd7a928f75aa4aaf3b412806a8c5ec2e22bae4985ec0 WatchSource:0}: Error finding container 1030fc50781f10a23d9ecd7a928f75aa4aaf3b412806a8c5ec2e22bae4985ec0: Status 404 returned error can't find the container with id 1030fc50781f10a23d9ecd7a928f75aa4aaf3b412806a8c5ec2e22bae4985ec0 Sep 16 18:55:56.214805 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:56.214784 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:55:56.352179 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:56.352135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" event={"ID":"767daf61-7d49-4541-9057-b60f19d2d7e4","Type":"ContainerStarted","Data":"1030fc50781f10a23d9ecd7a928f75aa4aaf3b412806a8c5ec2e22bae4985ec0"} Sep 16 18:55:58.361960 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:58.361921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" event={"ID":"767daf61-7d49-4541-9057-b60f19d2d7e4","Type":"ContainerStarted","Data":"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a"} Sep 16 18:55:58.362423 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:58.362024 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:55:58.378818 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:55:58.378771 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podStartSLOduration=1.40108825 podStartE2EDuration="3.378756606s" totalImagesPullingTime="1.977668356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:58.377112675 +0000 UTC m=+687.542133013" watchObservedRunningTime="2026-09-16 18:55:58.378756606 +0000 UTC m=+687.543776947" Sep 16 18:56:04.371858 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:04.371826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:56:05.677516 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:05.677480 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:56:05.677980 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:05.677698 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" containerID="cri-o://91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a" gracePeriod=30 Sep 16 18:56:09.370716 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:09.370673 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:14.370274 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:14.370235 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:19.370463 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:19.370425 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:19.370910 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:19.370537 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:56:24.370872 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:24.370784 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:29.370796 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:29.370752 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:34.370189 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:34.370142 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:35.554794 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.554746 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:56:35.561485 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.561442 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.563842 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.563811 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Sep 16 18:56:35.564002 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.563979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:56:35.564286 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.564274 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Sep 16 18:56:35.659333 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.659285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.659503 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.659372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.760005 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.759906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.760281 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.760246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.760695 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.760672 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.762915 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.762885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls\") pod \"model-chainer-648f748b4b-pjqp5\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.822558 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.822534 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:56:35.872741 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.872713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:35.962819 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.962789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle\") pod \"767daf61-7d49-4541-9057-b60f19d2d7e4\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " Sep 16 18:56:35.962970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.962836 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls\") pod \"767daf61-7d49-4541-9057-b60f19d2d7e4\" (UID: \"767daf61-7d49-4541-9057-b60f19d2d7e4\") " Sep 16 18:56:35.963218 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.963192 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "767daf61-7d49-4541-9057-b60f19d2d7e4" (UID: "767daf61-7d49-4541-9057-b60f19d2d7e4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:56:35.964914 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:35.964897 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "767daf61-7d49-4541-9057-b60f19d2d7e4" (UID: "767daf61-7d49-4541-9057-b60f19d2d7e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:56:36.064431 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.064399 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767daf61-7d49-4541-9057-b60f19d2d7e4-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:56:36.064431 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.064427 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767daf61-7d49-4541-9057-b60f19d2d7e4-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:56:36.147068 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.147043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:56:36.149603 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:56:36.149572 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3849260_af9d_4472_94b7_6b8834dd136b.slice/crio-538bccbf048a01b2434f81a3b4f441d90f47ac7412940a44d462db7f02c1e357 WatchSource:0}: Error finding container 538bccbf048a01b2434f81a3b4f441d90f47ac7412940a44d462db7f02c1e357: Status 404 returned error can't find the container with id 538bccbf048a01b2434f81a3b4f441d90f47ac7412940a44d462db7f02c1e357 Sep 16 18:56:36.532904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.532797 2559 generic.go:356] "Generic (PLEG): container finished" podID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerID="91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a" exitCode=0 Sep 16 18:56:36.532904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.532842 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" Sep 16 18:56:36.532904 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.532846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" event={"ID":"767daf61-7d49-4541-9057-b60f19d2d7e4","Type":"ContainerDied","Data":"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a"} Sep 16 18:56:36.533191 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.532916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr" event={"ID":"767daf61-7d49-4541-9057-b60f19d2d7e4","Type":"ContainerDied","Data":"1030fc50781f10a23d9ecd7a928f75aa4aaf3b412806a8c5ec2e22bae4985ec0"} Sep 16 18:56:36.533191 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.532948 2559 scope.go:117] "RemoveContainer" containerID="91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a" Sep 16 18:56:36.534682 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.534657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" event={"ID":"a3849260-af9d-4472-94b7-6b8834dd136b","Type":"ContainerStarted","Data":"54245c4f74e67ca319fa2331b49c7da041beee46ca2fad900816a848b01276da"} Sep 16 18:56:36.534774 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.534712 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:36.534774 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.534736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" event={"ID":"a3849260-af9d-4472-94b7-6b8834dd136b","Type":"ContainerStarted","Data":"538bccbf048a01b2434f81a3b4f441d90f47ac7412940a44d462db7f02c1e357"} Sep 16 18:56:36.542464 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.542449 2559 scope.go:117] "RemoveContainer" containerID="91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a" Sep 16 18:56:36.542691 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:56:36.542674 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a\": container with ID starting with 91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a not found: ID does not exist" containerID="91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a" Sep 16 18:56:36.542738 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.542697 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a"} err="failed to get container status \"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a\": rpc error: code = NotFound desc = could not find container \"91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a\": container with ID starting with 91893806bd8c118b9f8edb947570c86a4ab3cd8a9d45ac1d43daf147d012106a not found: ID does not exist" Sep 16 18:56:36.553292 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.553246 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podStartSLOduration=1.553233619 podStartE2EDuration="1.553233619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:36.552091487 +0000 UTC m=+725.717111830" watchObservedRunningTime="2026-09-16 18:56:36.553233619 +0000 UTC m=+725.718254004" Sep 16 18:56:36.564422 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.564394 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:56:36.566478 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:36.566459 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-aa795-cf8569c7b-m8kxr"] Sep 16 18:56:37.354513 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:37.354481 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" path="/var/lib/kubelet/pods/767daf61-7d49-4541-9057-b60f19d2d7e4/volumes" Sep 16 18:56:42.547369 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:42.547340 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:56:45.659239 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:45.659208 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:56:45.659626 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:45.659409 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" containerID="cri-o://54245c4f74e67ca319fa2331b49c7da041beee46ca2fad900816a848b01276da" gracePeriod=30 Sep 16 18:56:47.545785 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:47.545743 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:52.545623 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:52.545583 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:57.545131 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:57.545084 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:57.545598 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:56:57.545186 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:57:02.545047 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:02.545003 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:05.938861 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.938829 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 18:57:05.939235 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.939201 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" Sep 16 18:57:05.939235 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.939213 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" Sep 16 18:57:05.939310 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.939287 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="767daf61-7d49-4541-9057-b60f19d2d7e4" containerName="switch-graph-aa795" Sep 16 18:57:05.942162 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.942147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:05.944970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.944951 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eb09c-serving-cert\"" Sep 16 18:57:05.945392 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.945377 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eb09c-kube-rbac-proxy-sar-config\"" Sep 16 18:57:05.955338 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:05.955305 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 18:57:06.009590 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.009564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.009719 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.009643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.110362 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.110310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.110515 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.110420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.110515 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:57:06.110428 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-eb09c-serving-cert: secret "switch-graph-eb09c-serving-cert" not found Sep 16 18:57:06.110515 ip-10-0-129-152 kubenswrapper[2559]: E0916 18:57:06.110492 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls podName:feeb6a12-6d98-4ba8-8358-7955557a62b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:06.610472059 +0000 UTC m=+755.775492386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls") pod "switch-graph-eb09c-59454c5945-29wwk" (UID: "feeb6a12-6d98-4ba8-8358-7955557a62b1") : secret "switch-graph-eb09c-serving-cert" not found Sep 16 18:57:06.110989 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.110972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.615625 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.615592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.617985 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.617965 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") pod \"switch-graph-eb09c-59454c5945-29wwk\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:06.852398 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:06.852363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:07.121364 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.121345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 18:57:07.123801 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:57:07.123777 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeeb6a12_6d98_4ba8_8358_7955557a62b1.slice/crio-8c6f389c79e904b7826956329fe226f041909ac0fbd031f9d146a180e65ef840 WatchSource:0}: Error finding container 8c6f389c79e904b7826956329fe226f041909ac0fbd031f9d146a180e65ef840: Status 404 returned error can't find the container with id 8c6f389c79e904b7826956329fe226f041909ac0fbd031f9d146a180e65ef840 Sep 16 18:57:07.545946 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.545909 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:07.676168 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.676134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" event={"ID":"feeb6a12-6d98-4ba8-8358-7955557a62b1","Type":"ContainerStarted","Data":"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97"} Sep 16 18:57:07.676407 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.676184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" event={"ID":"feeb6a12-6d98-4ba8-8358-7955557a62b1","Type":"ContainerStarted","Data":"8c6f389c79e904b7826956329fe226f041909ac0fbd031f9d146a180e65ef840"} Sep 16 18:57:07.676407 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.676276 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:07.694599 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:07.694553 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podStartSLOduration=2.694538514 podStartE2EDuration="2.694538514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:07.69259883 +0000 UTC m=+756.857619185" watchObservedRunningTime="2026-09-16 18:57:07.694538514 +0000 UTC m=+756.859558855" Sep 16 18:57:12.545732 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:12.545697 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:13.686784 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:13.686754 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 18:57:15.716041 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.716013 2559 generic.go:356] "Generic (PLEG): container finished" podID="a3849260-af9d-4472-94b7-6b8834dd136b" containerID="54245c4f74e67ca319fa2331b49c7da041beee46ca2fad900816a848b01276da" exitCode=137 Sep 16 18:57:15.716405 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.716069 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" event={"ID":"a3849260-af9d-4472-94b7-6b8834dd136b","Type":"ContainerDied","Data":"54245c4f74e67ca319fa2331b49c7da041beee46ca2fad900816a848b01276da"} Sep 16 18:57:15.844540 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.844518 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:57:15.892284 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.892259 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls\") pod \"a3849260-af9d-4472-94b7-6b8834dd136b\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " Sep 16 18:57:15.892457 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.892313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle\") pod \"a3849260-af9d-4472-94b7-6b8834dd136b\" (UID: \"a3849260-af9d-4472-94b7-6b8834dd136b\") " Sep 16 18:57:15.892709 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.892687 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a3849260-af9d-4472-94b7-6b8834dd136b" (UID: "a3849260-af9d-4472-94b7-6b8834dd136b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:57:15.894285 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.894265 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3849260-af9d-4472-94b7-6b8834dd136b" (UID: "a3849260-af9d-4472-94b7-6b8834dd136b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:57:15.993675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.993609 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3849260-af9d-4472-94b7-6b8834dd136b-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:57:15.993675 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:15.993633 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3849260-af9d-4472-94b7-6b8834dd136b-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 18:57:16.722066 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:16.722023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" event={"ID":"a3849260-af9d-4472-94b7-6b8834dd136b","Type":"ContainerDied","Data":"538bccbf048a01b2434f81a3b4f441d90f47ac7412940a44d462db7f02c1e357"} Sep 16 18:57:16.722473 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:16.722085 2559 scope.go:117] "RemoveContainer" containerID="54245c4f74e67ca319fa2331b49c7da041beee46ca2fad900816a848b01276da" Sep 16 18:57:16.722473 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:16.722043 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5" Sep 16 18:57:16.745991 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:16.745968 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:57:16.754218 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:16.754196 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-648f748b4b-pjqp5"] Sep 16 18:57:17.354458 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:17.354427 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" path="/var/lib/kubelet/pods/a3849260-af9d-4472-94b7-6b8834dd136b/volumes" Sep 16 18:57:45.889527 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.889494 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 18:57:45.889970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.889849 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" Sep 16 18:57:45.889970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.889859 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" Sep 16 18:57:45.889970 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.889914 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3849260-af9d-4472-94b7-6b8834dd136b" containerName="model-chainer" Sep 16 18:57:45.894357 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.894341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:45.897189 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.897174 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2796f-kube-rbac-proxy-sar-config\"" Sep 16 18:57:45.897306 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.897287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2796f-serving-cert\"" Sep 16 18:57:45.911263 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:45.911243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 18:57:46.035334 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.035296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.035476 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.035381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.136126 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.136097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.136287 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.136149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.136753 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.136726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.138548 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.138525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls\") pod \"sequence-graph-2796f-5db7b6657f-b9tqk\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.204469 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.204407 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.473354 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.473254 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 18:57:46.476587 ip-10-0-129-152 kubenswrapper[2559]: W0916 18:57:46.476562 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e662bf_694b_4262_9251_02a95f9f067d.slice/crio-0fc81441e5b8ff0aaa554c0fcf6ff5bf4f132993ce36a1828c6126c5e9e5f465 WatchSource:0}: Error finding container 0fc81441e5b8ff0aaa554c0fcf6ff5bf4f132993ce36a1828c6126c5e9e5f465: Status 404 returned error can't find the container with id 0fc81441e5b8ff0aaa554c0fcf6ff5bf4f132993ce36a1828c6126c5e9e5f465 Sep 16 18:57:46.858466 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.858429 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" event={"ID":"a7e662bf-694b-4262-9251-02a95f9f067d","Type":"ContainerStarted","Data":"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379"} Sep 16 18:57:46.858644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.858485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" event={"ID":"a7e662bf-694b-4262-9251-02a95f9f067d","Type":"ContainerStarted","Data":"0fc81441e5b8ff0aaa554c0fcf6ff5bf4f132993ce36a1828c6126c5e9e5f465"} Sep 16 18:57:46.858644 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.858527 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:57:46.874657 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:46.874615 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podStartSLOduration=1.8746016829999999 podStartE2EDuration="1.874601683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:46.874387269 +0000 UTC m=+796.039407613" watchObservedRunningTime="2026-09-16 18:57:46.874601683 +0000 UTC m=+796.039622057" Sep 16 18:57:52.869569 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:57:52.869537 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 18:59:31.327446 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:59:31.327417 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 18:59:31.327909 ip-10-0-129-152 kubenswrapper[2559]: I0916 18:59:31.327765 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:04:31.359369 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:04:31.359337 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:04:31.360666 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:04:31.360645 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:05:20.489852 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:20.489822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 19:05:20.490277 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:20.489998 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" containerID="cri-o://4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97" gracePeriod=30 Sep 16 19:05:23.686022 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:23.685980 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:28.685592 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:28.685553 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:33.686124 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:33.686081 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:33.686527 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:33.686193 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 19:05:38.685875 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:38.685837 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:43.685500 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:43.685457 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:48.685466 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:48.685428 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:05:50.631105 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.631082 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 19:05:50.803032 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.802994 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") pod \"feeb6a12-6d98-4ba8-8358-7955557a62b1\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " Sep 16 19:05:50.803219 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.803140 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle\") pod \"feeb6a12-6d98-4ba8-8358-7955557a62b1\" (UID: \"feeb6a12-6d98-4ba8-8358-7955557a62b1\") " Sep 16 19:05:50.803516 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.803491 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "feeb6a12-6d98-4ba8-8358-7955557a62b1" (UID: "feeb6a12-6d98-4ba8-8358-7955557a62b1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:05:50.805199 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.805170 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feeb6a12-6d98-4ba8-8358-7955557a62b1" (UID: "feeb6a12-6d98-4ba8-8358-7955557a62b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:05:50.904415 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.904379 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feeb6a12-6d98-4ba8-8358-7955557a62b1-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:05:50.904415 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:50.904410 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feeb6a12-6d98-4ba8-8358-7955557a62b1-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:05:51.109552 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.109454 2559 generic.go:356] "Generic (PLEG): container finished" podID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerID="4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97" exitCode=0 Sep 16 19:05:51.109552 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.109509 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" Sep 16 19:05:51.109552 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.109536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" event={"ID":"feeb6a12-6d98-4ba8-8358-7955557a62b1","Type":"ContainerDied","Data":"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97"} Sep 16 19:05:51.109894 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.109613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk" event={"ID":"feeb6a12-6d98-4ba8-8358-7955557a62b1","Type":"ContainerDied","Data":"8c6f389c79e904b7826956329fe226f041909ac0fbd031f9d146a180e65ef840"} Sep 16 19:05:51.109894 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.109635 2559 scope.go:117] "RemoveContainer" containerID="4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97" Sep 16 19:05:51.118389 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.118369 2559 scope.go:117] "RemoveContainer" containerID="4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97" Sep 16 19:05:51.118636 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:05:51.118616 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97\": container with ID starting with 4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97 not found: ID does not exist" containerID="4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97" Sep 16 19:05:51.118702 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.118648 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97"} err="failed to get container status \"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97\": rpc error: code = NotFound desc = could not find container \"4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97\": container with ID starting with 4be110ff39d45f6e3ae9e6bc602a35384a9da1c4d14d347f8a5fa85743d57a97 not found: ID does not exist" Sep 16 19:05:51.130883 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.130855 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 19:05:51.137040 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.137021 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb09c-59454c5945-29wwk"] Sep 16 19:05:51.354909 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:05:51.354877 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" path="/var/lib/kubelet/pods/feeb6a12-6d98-4ba8-8358-7955557a62b1/volumes" Sep 16 19:06:00.489587 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:00.489553 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 19:06:00.490030 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:00.489752 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" containerID="cri-o://7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379" gracePeriod=30 Sep 16 19:06:02.867450 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:02.867413 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:07.867503 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:07.867466 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:12.868035 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:12.867990 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:12.868579 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:12.868099 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 19:06:17.868138 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:17.868101 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:20.767917 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.767885 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:06:20.768388 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.768242 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" Sep 16 19:06:20.768388 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.768254 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" Sep 16 19:06:20.768388 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.768310 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="feeb6a12-6d98-4ba8-8358-7955557a62b1" containerName="switch-graph-eb09c" Sep 16 19:06:20.771227 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.771213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:20.773636 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.773608 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c6de0-kube-rbac-proxy-sar-config\"" Sep 16 19:06:20.773636 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.773610 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c6de0-serving-cert\"" Sep 16 19:06:20.784127 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.784106 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:06:20.853435 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.853396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:20.853603 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.853448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:20.954443 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.954410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:20.954605 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.954454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:20.954605 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:06:20.954573 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-c6de0-serving-cert: secret "ensemble-graph-c6de0-serving-cert" not found Sep 16 19:06:20.954679 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:06:20.954624 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls podName:6a0f5bba-2715-4664-b25b-d9574da30905 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:21.454609006 +0000 UTC m=+1310.619629326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls") pod "ensemble-graph-c6de0-db6f655-4zjm6" (UID: "6a0f5bba-2715-4664-b25b-d9574da30905") : secret "ensemble-graph-c6de0-serving-cert" not found Sep 16 19:06:20.955034 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:20.955012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:21.458806 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:21.458751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:21.459031 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:06:21.458924 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-c6de0-serving-cert: secret "ensemble-graph-c6de0-serving-cert" not found Sep 16 19:06:21.459031 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:06:21.458997 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls podName:6a0f5bba-2715-4664-b25b-d9574da30905 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:22.458978882 +0000 UTC m=+1311.623999216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls") pod "ensemble-graph-c6de0-db6f655-4zjm6" (UID: "6a0f5bba-2715-4664-b25b-d9574da30905") : secret "ensemble-graph-c6de0-serving-cert" not found Sep 16 19:06:22.467956 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:22.467918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:22.470383 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:22.470359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"ensemble-graph-c6de0-db6f655-4zjm6\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:22.581370 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:22.581312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:22.851171 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:22.851143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:06:22.853472 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:06:22.853442 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0f5bba_2715_4664_b25b_d9574da30905.slice/crio-e2fc8ad2ff37be4fe6378cf590f952e8df11c03d9e147e2a51bf6e679d5af765 WatchSource:0}: Error finding container e2fc8ad2ff37be4fe6378cf590f952e8df11c03d9e147e2a51bf6e679d5af765: Status 404 returned error can't find the container with id e2fc8ad2ff37be4fe6378cf590f952e8df11c03d9e147e2a51bf6e679d5af765 Sep 16 19:06:22.867966 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:22.867935 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:23.266013 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:23.265915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" event={"ID":"6a0f5bba-2715-4664-b25b-d9574da30905","Type":"ContainerStarted","Data":"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2"} Sep 16 19:06:23.266013 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:23.265986 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:23.266013 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:23.266011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" event={"ID":"6a0f5bba-2715-4664-b25b-d9574da30905","Type":"ContainerStarted","Data":"e2fc8ad2ff37be4fe6378cf590f952e8df11c03d9e147e2a51bf6e679d5af765"} Sep 16 19:06:23.285917 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:23.285857 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podStartSLOduration=3.285837273 podStartE2EDuration="3.285837273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:23.285039855 +0000 UTC m=+1312.450060198" watchObservedRunningTime="2026-09-16 19:06:23.285837273 +0000 UTC m=+1312.450857619" Sep 16 19:06:27.867357 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:27.867307 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:29.276783 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:29.276752 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:30.679598 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.679575 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 19:06:30.737856 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.737825 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle\") pod \"a7e662bf-694b-4262-9251-02a95f9f067d\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " Sep 16 19:06:30.737984 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.737881 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls\") pod \"a7e662bf-694b-4262-9251-02a95f9f067d\" (UID: \"a7e662bf-694b-4262-9251-02a95f9f067d\") " Sep 16 19:06:30.738137 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.738116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a7e662bf-694b-4262-9251-02a95f9f067d" (UID: "a7e662bf-694b-4262-9251-02a95f9f067d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:06:30.739948 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.739921 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7e662bf-694b-4262-9251-02a95f9f067d" (UID: "a7e662bf-694b-4262-9251-02a95f9f067d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:06:30.839536 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.839500 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7e662bf-694b-4262-9251-02a95f9f067d-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:06:30.839536 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.839533 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7e662bf-694b-4262-9251-02a95f9f067d-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:06:30.842547 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.842518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:06:30.842726 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:30.842687 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" containerID="cri-o://a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2" gracePeriod=30 Sep 16 19:06:31.301493 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.301461 2559 generic.go:356] "Generic (PLEG): container finished" podID="a7e662bf-694b-4262-9251-02a95f9f067d" containerID="7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379" exitCode=0 Sep 16 19:06:31.301663 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.301506 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" Sep 16 19:06:31.301663 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.301542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" event={"ID":"a7e662bf-694b-4262-9251-02a95f9f067d","Type":"ContainerDied","Data":"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379"} Sep 16 19:06:31.301663 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.301594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk" event={"ID":"a7e662bf-694b-4262-9251-02a95f9f067d","Type":"ContainerDied","Data":"0fc81441e5b8ff0aaa554c0fcf6ff5bf4f132993ce36a1828c6126c5e9e5f465"} Sep 16 19:06:31.301663 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.301618 2559 scope.go:117] "RemoveContainer" containerID="7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379" Sep 16 19:06:31.310691 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.310674 2559 scope.go:117] "RemoveContainer" containerID="7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379" Sep 16 19:06:31.310943 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:06:31.310926 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379\": container with ID starting with 7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379 not found: ID does not exist" containerID="7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379" Sep 16 19:06:31.310991 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.310950 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379"} err="failed to get container status \"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379\": rpc error: code = NotFound desc = could not find container \"7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379\": container with ID starting with 7ff5ddc5696e409327470bb1983a803ca33b5b71713a3c0118ba0e1c712e3379 not found: ID does not exist" Sep 16 19:06:31.325916 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.325894 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 19:06:31.330251 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.330231 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2796f-5db7b6657f-b9tqk"] Sep 16 19:06:31.353724 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:31.353702 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" path="/var/lib/kubelet/pods/a7e662bf-694b-4262-9251-02a95f9f067d/volumes" Sep 16 19:06:34.274417 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:34.274380 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:39.274678 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:39.274634 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:44.274978 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:44.274936 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:44.275393 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:44.275041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:06:49.275209 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:49.275165 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:54.274217 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:54.274128 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:06:59.274794 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:06:59.274751 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:00.729830 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.729793 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:00.730219 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.730148 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" Sep 16 19:07:00.730219 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.730160 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" Sep 16 19:07:00.730300 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.730221 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e662bf-694b-4262-9251-02a95f9f067d" containerName="sequence-graph-2796f" Sep 16 19:07:00.734689 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.734671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.737524 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.737495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f64b9-serving-cert\"" Sep 16 19:07:00.737971 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.737949 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f64b9-kube-rbac-proxy-sar-config\"" Sep 16 19:07:00.744142 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.744116 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:00.790165 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.790137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.790304 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.790187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.890851 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.890820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.891026 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.890916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.891026 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:07:00.890973 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f64b9-serving-cert: secret "sequence-graph-f64b9-serving-cert" not found Sep 16 19:07:00.891149 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:07:00.891039 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls podName:14a6323e-d9a4-42ed-b646-93dc7db72be2 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.391023786 +0000 UTC m=+1350.556044121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls") pod "sequence-graph-f64b9-59cc8f9984-ssqv5" (UID: "14a6323e-d9a4-42ed-b646-93dc7db72be2") : secret "sequence-graph-f64b9-serving-cert" not found Sep 16 19:07:00.891583 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.891560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:00.984835 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:00.984778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:07:01.093178 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.093144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle\") pod \"6a0f5bba-2715-4664-b25b-d9574da30905\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " Sep 16 19:07:01.093380 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.093232 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") pod \"6a0f5bba-2715-4664-b25b-d9574da30905\" (UID: \"6a0f5bba-2715-4664-b25b-d9574da30905\") " Sep 16 19:07:01.093578 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.093556 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6a0f5bba-2715-4664-b25b-d9574da30905" (UID: "6a0f5bba-2715-4664-b25b-d9574da30905"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:01.095239 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.095207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a0f5bba-2715-4664-b25b-d9574da30905" (UID: "6a0f5bba-2715-4664-b25b-d9574da30905"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:01.194442 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.194413 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a0f5bba-2715-4664-b25b-d9574da30905-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:07:01.194442 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.194438 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f5bba-2715-4664-b25b-d9574da30905-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:07:01.400337 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.395943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:01.400337 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.399506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") pod \"sequence-graph-f64b9-59cc8f9984-ssqv5\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:01.438033 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.438002 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a0f5bba-2715-4664-b25b-d9574da30905" containerID="a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2" exitCode=0 Sep 16 19:07:01.438194 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.438060 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" Sep 16 19:07:01.438194 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.438058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" event={"ID":"6a0f5bba-2715-4664-b25b-d9574da30905","Type":"ContainerDied","Data":"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2"} Sep 16 19:07:01.438194 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.438152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6" event={"ID":"6a0f5bba-2715-4664-b25b-d9574da30905","Type":"ContainerDied","Data":"e2fc8ad2ff37be4fe6378cf590f952e8df11c03d9e147e2a51bf6e679d5af765"} Sep 16 19:07:01.438194 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.438181 2559 scope.go:117] "RemoveContainer" containerID="a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2" Sep 16 19:07:01.446306 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.446286 2559 scope.go:117] "RemoveContainer" containerID="a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2" Sep 16 19:07:01.446593 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:07:01.446573 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2\": container with ID starting with a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2 not found: ID does not exist" containerID="a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2" Sep 16 19:07:01.446648 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.446600 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2"} err="failed to get container status \"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2\": rpc error: code = NotFound desc = could not find container \"a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2\": container with ID starting with a58985fbb18cf402520e6fb1e5afae0b1d4183b9ba40278bc9eede979d158ae2 not found: ID does not exist" Sep 16 19:07:01.467066 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.467042 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:07:01.480709 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.480688 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c6de0-db6f655-4zjm6"] Sep 16 19:07:01.644853 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.644775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:01.921208 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:01.921174 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:01.922479 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:07:01.922458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a6323e_d9a4_42ed_b646_93dc7db72be2.slice/crio-af5507a6231bbda584ea5e5f4c4c26b95452b2ee8f0ae5dd37b45004d53d5a4c WatchSource:0}: Error finding container af5507a6231bbda584ea5e5f4c4c26b95452b2ee8f0ae5dd37b45004d53d5a4c: Status 404 returned error can't find the container with id af5507a6231bbda584ea5e5f4c4c26b95452b2ee8f0ae5dd37b45004d53d5a4c Sep 16 19:07:02.444890 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:02.444842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" event={"ID":"14a6323e-d9a4-42ed-b646-93dc7db72be2","Type":"ContainerStarted","Data":"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726"} Sep 16 19:07:02.445090 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:02.444914 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" event={"ID":"14a6323e-d9a4-42ed-b646-93dc7db72be2","Type":"ContainerStarted","Data":"af5507a6231bbda584ea5e5f4c4c26b95452b2ee8f0ae5dd37b45004d53d5a4c"} Sep 16 19:07:02.445090 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:02.444947 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:02.463100 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:02.463052 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podStartSLOduration=2.463037727 podStartE2EDuration="2.463037727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:02.462528617 +0000 UTC m=+1351.627548960" watchObservedRunningTime="2026-09-16 19:07:02.463037727 +0000 UTC m=+1351.628058071" Sep 16 19:07:03.354289 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:03.354256 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" path="/var/lib/kubelet/pods/6a0f5bba-2715-4664-b25b-d9574da30905/volumes" Sep 16 19:07:08.456148 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:08.456121 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:10.807664 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:10.807633 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:10.808091 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:10.807810 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" containerID="cri-o://3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726" gracePeriod=30 Sep 16 19:07:13.454523 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:13.454488 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:18.454839 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:18.454795 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:23.455177 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:23.455138 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:23.455601 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:23.455251 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:28.462918 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:28.462492 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:31.199254 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.196577 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:07:31.199254 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.197429 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" Sep 16 19:07:31.199254 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.197458 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" Sep 16 19:07:31.199254 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.197658 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0f5bba-2715-4664-b25b-d9574da30905" containerName="ensemble-graph-c6de0" Sep 16 19:07:31.203379 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.203351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.205850 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.205833 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-a3448-serving-cert\"" Sep 16 19:07:31.205969 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.205938 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-a3448-kube-rbac-proxy-sar-config\"" Sep 16 19:07:31.210418 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.210399 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:07:31.352112 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.352070 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.352263 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.352246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.452940 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.452883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.452940 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.452925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.453704 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.453685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.455919 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.455903 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-a3448-serving-cert\"" Sep 16 19:07:31.465453 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.465433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls\") pod \"ensemble-graph-a3448-6fb64b7f46-nrcwp\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.513495 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.513471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:31.785713 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:31.785665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:07:31.787955 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:07:31.787929 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f08c98_48f8_485d_ac4f_004d99fb4972.slice/crio-7f40388d6f1e6cc3d968ba589cb82aaadb9264ebeb91759f97b13c874f6bbe19 WatchSource:0}: Error finding container 7f40388d6f1e6cc3d968ba589cb82aaadb9264ebeb91759f97b13c874f6bbe19: Status 404 returned error can't find the container with id 7f40388d6f1e6cc3d968ba589cb82aaadb9264ebeb91759f97b13c874f6bbe19 Sep 16 19:07:32.591253 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:32.591215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" event={"ID":"65f08c98-48f8-485d-ac4f-004d99fb4972","Type":"ContainerStarted","Data":"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858"} Sep 16 19:07:32.591637 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:32.591275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" event={"ID":"65f08c98-48f8-485d-ac4f-004d99fb4972","Type":"ContainerStarted","Data":"7f40388d6f1e6cc3d968ba589cb82aaadb9264ebeb91759f97b13c874f6bbe19"} Sep 16 19:07:32.591637 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:32.591368 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:32.611290 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:32.611245 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podStartSLOduration=1.6112315069999998 podStartE2EDuration="1.611231507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:32.609598924 +0000 UTC m=+1381.774619271" watchObservedRunningTime="2026-09-16 19:07:32.611231507 +0000 UTC m=+1381.776251850" Sep 16 19:07:33.454631 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:33.454591 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:38.454914 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:38.454873 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:38.601296 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:38.601260 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:07:40.954676 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:40.954654 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:41.134908 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.134820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") pod \"14a6323e-d9a4-42ed-b646-93dc7db72be2\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " Sep 16 19:07:41.134908 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.134900 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle\") pod \"14a6323e-d9a4-42ed-b646-93dc7db72be2\" (UID: \"14a6323e-d9a4-42ed-b646-93dc7db72be2\") " Sep 16 19:07:41.135236 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.135214 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "14a6323e-d9a4-42ed-b646-93dc7db72be2" (UID: "14a6323e-d9a4-42ed-b646-93dc7db72be2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:41.136846 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.136824 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14a6323e-d9a4-42ed-b646-93dc7db72be2" (UID: "14a6323e-d9a4-42ed-b646-93dc7db72be2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:41.235743 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.235715 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a6323e-d9a4-42ed-b646-93dc7db72be2-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:07:41.235743 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.235740 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a6323e-d9a4-42ed-b646-93dc7db72be2-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:07:41.640806 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.640774 2559 generic.go:356] "Generic (PLEG): container finished" podID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerID="3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726" exitCode=0 Sep 16 19:07:41.640976 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.640814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" event={"ID":"14a6323e-d9a4-42ed-b646-93dc7db72be2","Type":"ContainerDied","Data":"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726"} Sep 16 19:07:41.640976 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.640850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" event={"ID":"14a6323e-d9a4-42ed-b646-93dc7db72be2","Type":"ContainerDied","Data":"af5507a6231bbda584ea5e5f4c4c26b95452b2ee8f0ae5dd37b45004d53d5a4c"} Sep 16 19:07:41.640976 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.640878 2559 scope.go:117] "RemoveContainer" containerID="3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726" Sep 16 19:07:41.640976 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.640816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5" Sep 16 19:07:41.650112 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.650081 2559 scope.go:117] "RemoveContainer" containerID="3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726" Sep 16 19:07:41.650344 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:07:41.650306 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726\": container with ID starting with 3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726 not found: ID does not exist" containerID="3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726" Sep 16 19:07:41.650400 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.650356 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726"} err="failed to get container status \"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726\": rpc error: code = NotFound desc = could not find container \"3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726\": container with ID starting with 3dd96e50e6f4eb7b50d751cc6cfb012c50b11947901fa6e68b281a2e96f65726 not found: ID does not exist" Sep 16 19:07:41.658442 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.658419 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:41.665349 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:41.665024 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f64b9-59cc8f9984-ssqv5"] Sep 16 19:07:43.354018 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:07:43.353986 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" path="/var/lib/kubelet/pods/14a6323e-d9a4-42ed-b646-93dc7db72be2/volumes" Sep 16 19:08:11.186255 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.186216 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:08:11.186729 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.186617 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" Sep 16 19:08:11.186729 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.186630 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" Sep 16 19:08:11.186729 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.186701 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a6323e-d9a4-42ed-b646-93dc7db72be2" containerName="sequence-graph-f64b9" Sep 16 19:08:11.189679 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.189663 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.192177 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.192155 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e99c4-kube-rbac-proxy-sar-config\"" Sep 16 19:08:11.192625 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.192608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e99c4-serving-cert\"" Sep 16 19:08:11.207143 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.207117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:08:11.286138 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.286107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.286289 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.286171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.386812 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.386787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.386958 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.386841 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.387579 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.387557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.389233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.389207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls\") pod \"sequence-graph-e99c4-95c6d94f8-tgdbt\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.499824 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.499745 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:11.769960 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.769933 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:08:11.772576 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:08:11.772553 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe0ed08_8180_48ca_a889_ac126d64bf25.slice/crio-08762c30de50b1cd90292623fe4515a63c146a0d25c67825eef4b3681d0c565b WatchSource:0}: Error finding container 08762c30de50b1cd90292623fe4515a63c146a0d25c67825eef4b3681d0c565b: Status 404 returned error can't find the container with id 08762c30de50b1cd90292623fe4515a63c146a0d25c67825eef4b3681d0c565b Sep 16 19:08:11.786852 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:11.786828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" event={"ID":"ebe0ed08-8180-48ca-a889-ac126d64bf25","Type":"ContainerStarted","Data":"08762c30de50b1cd90292623fe4515a63c146a0d25c67825eef4b3681d0c565b"} Sep 16 19:08:12.792993 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:12.792953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" event={"ID":"ebe0ed08-8180-48ca-a889-ac126d64bf25","Type":"ContainerStarted","Data":"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18"} Sep 16 19:08:12.793469 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:12.793013 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:08:12.811772 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:12.811725 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podStartSLOduration=1.811711876 podStartE2EDuration="1.811711876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:12.809841118 +0000 UTC m=+1421.974861460" watchObservedRunningTime="2026-09-16 19:08:12.811711876 +0000 UTC m=+1421.976732267" Sep 16 19:08:18.803656 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:08:18.803594 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:09:31.394407 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:09:31.394381 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:09:31.395257 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:09:31.395237 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:14:31.426043 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:14:31.426016 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:14:31.428447 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:14:31.428425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:15:45.806239 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:45.806204 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:15:45.808680 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:45.806431 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" containerID="cri-o://6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858" gracePeriod=30 Sep 16 19:15:48.599404 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:48.599364 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:15:53.599633 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:53.599591 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:15:58.600086 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:58.600050 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:15:58.600480 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:15:58.600160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:16:03.599933 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:03.599891 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:08.600032 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:08.599996 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:13.600128 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:13.600087 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:15.956291 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:15.956264 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:16:16.005453 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.005416 2559 generic.go:356] "Generic (PLEG): container finished" podID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerID="6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858" exitCode=0 Sep 16 19:16:16.005576 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.005463 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" Sep 16 19:16:16.005576 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.005467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" event={"ID":"65f08c98-48f8-485d-ac4f-004d99fb4972","Type":"ContainerDied","Data":"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858"} Sep 16 19:16:16.005576 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.005569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp" event={"ID":"65f08c98-48f8-485d-ac4f-004d99fb4972","Type":"ContainerDied","Data":"7f40388d6f1e6cc3d968ba589cb82aaadb9264ebeb91759f97b13c874f6bbe19"} Sep 16 19:16:16.005687 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.005593 2559 scope.go:117] "RemoveContainer" containerID="6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858" Sep 16 19:16:16.013715 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.013684 2559 scope.go:117] "RemoveContainer" containerID="6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858" Sep 16 19:16:16.013951 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:16:16.013933 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858\": container with ID starting with 6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858 not found: ID does not exist" containerID="6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858" Sep 16 19:16:16.013995 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.013962 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858"} err="failed to get container status \"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858\": rpc error: code = NotFound desc = could not find container \"6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858\": container with ID starting with 6220cbb37e41492adb483311d63132232ead6fc57956ca97902af06874569858 not found: ID does not exist" Sep 16 19:16:16.026693 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.026671 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle\") pod \"65f08c98-48f8-485d-ac4f-004d99fb4972\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " Sep 16 19:16:16.026758 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.026733 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls\") pod \"65f08c98-48f8-485d-ac4f-004d99fb4972\" (UID: \"65f08c98-48f8-485d-ac4f-004d99fb4972\") " Sep 16 19:16:16.027081 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.027060 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "65f08c98-48f8-485d-ac4f-004d99fb4972" (UID: "65f08c98-48f8-485d-ac4f-004d99fb4972"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:16:16.028587 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.028566 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65f08c98-48f8-485d-ac4f-004d99fb4972" (UID: "65f08c98-48f8-485d-ac4f-004d99fb4972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:16:16.128003 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.127948 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f08c98-48f8-485d-ac4f-004d99fb4972-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:16:16.128003 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.127968 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f08c98-48f8-485d-ac4f-004d99fb4972-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:16:16.328266 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.328235 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:16:16.334682 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:16.334660 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-a3448-6fb64b7f46-nrcwp"] Sep 16 19:16:17.354029 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:17.353996 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" path="/var/lib/kubelet/pods/65f08c98-48f8-485d-ac4f-004d99fb4972/volumes" Sep 16 19:16:25.641688 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:25.641657 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:16:25.642144 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:25.641848 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" containerID="cri-o://6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18" gracePeriod=30 Sep 16 19:16:28.801467 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:28.801430 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:33.802170 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:33.802125 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:38.801764 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:38.801723 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:38.802173 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:38.801841 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:16:43.801356 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:43.801299 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:46.072660 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.072625 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:16:46.073072 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.073005 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" Sep 16 19:16:46.073072 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.073019 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" Sep 16 19:16:46.073154 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.073095 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f08c98-48f8-485d-ac4f-004d99fb4972" containerName="ensemble-graph-a3448" Sep 16 19:16:46.076052 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.076038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.078662 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.078607 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-da3eb-serving-cert\"" Sep 16 19:16:46.078662 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.078642 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-da3eb-kube-rbac-proxy-sar-config\"" Sep 16 19:16:46.089745 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.089719 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:16:46.179451 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.179424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.179590 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.179473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.280348 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.280302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.280482 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.280407 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.280955 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.280930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.282813 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.282784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls\") pod \"splitter-graph-da3eb-5b6db7747c-28d44\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.386253 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.386199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:46.660107 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:16:46.660072 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec43c15_7101_4ecd_b712_19ef88f41b1b.slice/crio-86793c56fedabb70cc8e3c54814ea50a638d3d6c7d2af8747233e70b2d66aaa6 WatchSource:0}: Error finding container 86793c56fedabb70cc8e3c54814ea50a638d3d6c7d2af8747233e70b2d66aaa6: Status 404 returned error can't find the container with id 86793c56fedabb70cc8e3c54814ea50a638d3d6c7d2af8747233e70b2d66aaa6 Sep 16 19:16:46.660844 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:46.660823 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:16:47.149349 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:47.149281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" event={"ID":"2ec43c15-7101-4ecd-b712-19ef88f41b1b","Type":"ContainerStarted","Data":"8adf5297ee8b7e1aaa18c495152e1337baf61ff91b88920a56a8eeb5dce0464b"} Sep 16 19:16:47.149771 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:47.149401 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" event={"ID":"2ec43c15-7101-4ecd-b712-19ef88f41b1b","Type":"ContainerStarted","Data":"86793c56fedabb70cc8e3c54814ea50a638d3d6c7d2af8747233e70b2d66aaa6"} Sep 16 19:16:47.149771 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:47.149436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:47.223206 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:47.223159 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podStartSLOduration=1.223144253 podStartE2EDuration="1.223144253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:47.223010381 +0000 UTC m=+1936.388030724" watchObservedRunningTime="2026-09-16 19:16:47.223144253 +0000 UTC m=+1936.388164595" Sep 16 19:16:48.801247 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:48.801207 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:53.162426 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:53.162394 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:16:53.802112 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:53.802076 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:16:55.787875 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:55.787854 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:16:55.966386 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:55.966292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls\") pod \"ebe0ed08-8180-48ca-a889-ac126d64bf25\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " Sep 16 19:16:55.966386 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:55.966361 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle\") pod \"ebe0ed08-8180-48ca-a889-ac126d64bf25\" (UID: \"ebe0ed08-8180-48ca-a889-ac126d64bf25\") " Sep 16 19:16:55.966711 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:55.966688 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ebe0ed08-8180-48ca-a889-ac126d64bf25" (UID: "ebe0ed08-8180-48ca-a889-ac126d64bf25"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:16:55.968360 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:55.968336 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebe0ed08-8180-48ca-a889-ac126d64bf25" (UID: "ebe0ed08-8180-48ca-a889-ac126d64bf25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:16:56.067202 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.067177 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebe0ed08-8180-48ca-a889-ac126d64bf25-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:16:56.067202 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.067202 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebe0ed08-8180-48ca-a889-ac126d64bf25-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:16:56.118101 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.118073 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:16:56.118305 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.118284 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" containerID="cri-o://8adf5297ee8b7e1aaa18c495152e1337baf61ff91b88920a56a8eeb5dce0464b" gracePeriod=30 Sep 16 19:16:56.197932 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.197901 2559 generic.go:356] "Generic (PLEG): container finished" podID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerID="6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18" exitCode=0 Sep 16 19:16:56.198088 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.197939 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" Sep 16 19:16:56.198088 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.197979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" event={"ID":"ebe0ed08-8180-48ca-a889-ac126d64bf25","Type":"ContainerDied","Data":"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18"} Sep 16 19:16:56.198088 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.198024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt" event={"ID":"ebe0ed08-8180-48ca-a889-ac126d64bf25","Type":"ContainerDied","Data":"08762c30de50b1cd90292623fe4515a63c146a0d25c67825eef4b3681d0c565b"} Sep 16 19:16:56.198088 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.198043 2559 scope.go:117] "RemoveContainer" containerID="6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18" Sep 16 19:16:56.208998 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.208975 2559 scope.go:117] "RemoveContainer" containerID="6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18" Sep 16 19:16:56.209332 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:16:56.209288 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18\": container with ID starting with 6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18 not found: ID does not exist" containerID="6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18" Sep 16 19:16:56.209447 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.209345 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18"} err="failed to get container status \"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18\": rpc error: code = NotFound desc = could not find container \"6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18\": container with ID starting with 6a6e126a2be94d0ff48ce43f69fa8601a213c617eebcb70d4a8aef3c511b7a18 not found: ID does not exist" Sep 16 19:16:56.226811 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.226757 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:16:56.233681 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:56.233656 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e99c4-95c6d94f8-tgdbt"] Sep 16 19:16:57.354479 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:57.354448 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" path="/var/lib/kubelet/pods/ebe0ed08-8180-48ca-a889-ac126d64bf25/volumes" Sep 16 19:16:58.161300 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:16:58.161263 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:03.160949 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:03.160912 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:08.161226 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:08.161180 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:08.161758 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:08.161344 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:17:13.160923 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:13.160863 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:18.160766 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:18.160726 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:23.160960 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:23.160919 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:17:25.903069 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.902993 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:17:25.903504 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.903359 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" Sep 16 19:17:25.903504 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.903370 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" Sep 16 19:17:25.903504 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.903442 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe0ed08-8180-48ca-a889-ac126d64bf25" containerName="sequence-graph-e99c4" Sep 16 19:17:25.906486 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.906468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:25.909026 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.908996 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ca22e-serving-cert\"" Sep 16 19:17:25.909026 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.909009 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ca22e-kube-rbac-proxy-sar-config\"" Sep 16 19:17:25.915665 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:25.915641 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:17:26.015481 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.015457 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.015601 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.015488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.116462 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.116437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.116574 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.116466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.116630 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:17:26.116571 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-ca22e-serving-cert: secret "switch-graph-ca22e-serving-cert" not found Sep 16 19:17:26.116630 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:17:26.116626 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls podName:359602d5-6cff-45e4-9e3d-f500149e3eab nodeName:}" failed. No retries permitted until 2026-09-16 19:17:26.61661261 +0000 UTC m=+1975.781632935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls") pod "switch-graph-ca22e-64559f96c5-kscpd" (UID: "359602d5-6cff-45e4-9e3d-f500149e3eab") : secret "switch-graph-ca22e-serving-cert" not found Sep 16 19:17:26.117044 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.117027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.341361 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.341313 2559 generic.go:356] "Generic (PLEG): container finished" podID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerID="8adf5297ee8b7e1aaa18c495152e1337baf61ff91b88920a56a8eeb5dce0464b" exitCode=0 Sep 16 19:17:26.341519 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.341389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" event={"ID":"2ec43c15-7101-4ecd-b712-19ef88f41b1b","Type":"ContainerDied","Data":"8adf5297ee8b7e1aaa18c495152e1337baf61ff91b88920a56a8eeb5dce0464b"} Sep 16 19:17:26.621014 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.620911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.623335 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.623299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") pod \"switch-graph-ca22e-64559f96c5-kscpd\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.761095 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.761066 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:17:26.817092 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.817062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:26.821854 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.821835 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls\") pod \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " Sep 16 19:17:26.821933 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.821896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle\") pod \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\" (UID: \"2ec43c15-7101-4ecd-b712-19ef88f41b1b\") " Sep 16 19:17:26.822362 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.822316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2ec43c15-7101-4ecd-b712-19ef88f41b1b" (UID: "2ec43c15-7101-4ecd-b712-19ef88f41b1b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:17:26.823836 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.823807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ec43c15-7101-4ecd-b712-19ef88f41b1b" (UID: "2ec43c15-7101-4ecd-b712-19ef88f41b1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:17:26.923380 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.923293 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ec43c15-7101-4ecd-b712-19ef88f41b1b-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:17:26.923380 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:26.923339 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ec43c15-7101-4ecd-b712-19ef88f41b1b-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:17:27.089154 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.089111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:17:27.092083 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:17:27.092056 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359602d5_6cff_45e4_9e3d_f500149e3eab.slice/crio-1079738d9065e681b0660d30167360b24654d1ab087f247454295b9ae11a40dc WatchSource:0}: Error finding container 1079738d9065e681b0660d30167360b24654d1ab087f247454295b9ae11a40dc: Status 404 returned error can't find the container with id 1079738d9065e681b0660d30167360b24654d1ab087f247454295b9ae11a40dc Sep 16 19:17:27.347023 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.346991 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" Sep 16 19:17:27.347201 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.347046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44" event={"ID":"2ec43c15-7101-4ecd-b712-19ef88f41b1b","Type":"ContainerDied","Data":"86793c56fedabb70cc8e3c54814ea50a638d3d6c7d2af8747233e70b2d66aaa6"} Sep 16 19:17:27.347201 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.347115 2559 scope.go:117] "RemoveContainer" containerID="8adf5297ee8b7e1aaa18c495152e1337baf61ff91b88920a56a8eeb5dce0464b" Sep 16 19:17:27.354987 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.354963 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:27.355166 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.355010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" event={"ID":"359602d5-6cff-45e4-9e3d-f500149e3eab","Type":"ContainerStarted","Data":"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b"} Sep 16 19:17:27.355166 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.355039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" event={"ID":"359602d5-6cff-45e4-9e3d-f500149e3eab","Type":"ContainerStarted","Data":"1079738d9065e681b0660d30167360b24654d1ab087f247454295b9ae11a40dc"} Sep 16 19:17:27.370293 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.370252 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podStartSLOduration=2.370238534 podStartE2EDuration="2.370238534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:27.368066563 +0000 UTC m=+1976.533086907" watchObservedRunningTime="2026-09-16 19:17:27.370238534 +0000 UTC m=+1976.535258907" Sep 16 19:17:27.384998 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.384974 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:17:27.388679 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:27.388653 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-da3eb-5b6db7747c-28d44"] Sep 16 19:17:29.354300 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:29.354256 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" path="/var/lib/kubelet/pods/2ec43c15-7101-4ecd-b712-19ef88f41b1b/volumes" Sep 16 19:17:33.360766 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:33.360738 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:17:56.393439 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.393401 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:17:56.393831 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.393779 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" Sep 16 19:17:56.393831 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.393791 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" Sep 16 19:17:56.393911 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.393850 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec43c15-7101-4ecd-b712-19ef88f41b1b" containerName="splitter-graph-da3eb" Sep 16 19:17:56.396842 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.396825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:56.399273 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.399253 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-4a90a-serving-cert\"" Sep 16 19:17:56.399617 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.399601 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-4a90a-kube-rbac-proxy-sar-config\"" Sep 16 19:17:56.405100 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.405074 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:17:56.463513 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.463485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:56.463656 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.463532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:56.564656 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.564617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:56.564831 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.564716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:56.564831 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:17:56.564823 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-4a90a-serving-cert: secret "splitter-graph-4a90a-serving-cert" not found Sep 16 19:17:56.564927 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:17:56.564885 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls podName:3c7f2418-d595-47ab-8e5d-1373653804fd nodeName:}" failed. No retries permitted until 2026-09-16 19:17:57.064868785 +0000 UTC m=+2006.229889105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls") pod "splitter-graph-4a90a-5d7c796cdb-xsjm4" (UID: "3c7f2418-d595-47ab-8e5d-1373653804fd") : secret "splitter-graph-4a90a-serving-cert" not found Sep 16 19:17:56.565268 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:56.565251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:57.068648 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:57.068604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:57.071088 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:57.071061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") pod \"splitter-graph-4a90a-5d7c796cdb-xsjm4\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:57.308072 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:57.308031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:57.578868 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:57.578841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:17:57.581020 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:17:57.580998 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7f2418_d595_47ab_8e5d_1373653804fd.slice/crio-1828a69304a32a55b21cd5d0f1d765e22d78b95af058ee8bdd05e201d587a64b WatchSource:0}: Error finding container 1828a69304a32a55b21cd5d0f1d765e22d78b95af058ee8bdd05e201d587a64b: Status 404 returned error can't find the container with id 1828a69304a32a55b21cd5d0f1d765e22d78b95af058ee8bdd05e201d587a64b Sep 16 19:17:58.493641 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:58.493610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" event={"ID":"3c7f2418-d595-47ab-8e5d-1373653804fd","Type":"ContainerStarted","Data":"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790"} Sep 16 19:17:58.493798 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:58.493665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" event={"ID":"3c7f2418-d595-47ab-8e5d-1373653804fd","Type":"ContainerStarted","Data":"1828a69304a32a55b21cd5d0f1d765e22d78b95af058ee8bdd05e201d587a64b"} Sep 16 19:17:58.493798 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:58.493702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:17:58.510185 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:17:58.510141 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podStartSLOduration=2.5101272789999998 podStartE2EDuration="2.510127279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:58.509112735 +0000 UTC m=+2007.674133078" watchObservedRunningTime="2026-09-16 19:17:58.510127279 +0000 UTC m=+2007.675147621" Sep 16 19:18:04.504182 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:18:04.504152 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:19:31.455546 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:19:31.455515 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:19:31.459873 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:19:31.459839 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:24:31.485697 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:24:31.485659 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:24:31.490954 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:24:31.490928 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:26:10.962587 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:10.962550 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:26:10.963181 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:10.962751 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" containerID="cri-o://bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790" gracePeriod=30 Sep 16 19:26:14.502896 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:14.502856 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:19.502248 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:19.502209 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:24.502547 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:24.502462 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:24.502929 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:24.502561 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:26:29.502638 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:29.502597 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:34.502403 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:34.502363 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:39.502172 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:39.502135 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:26:41.103728 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.103702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:26:41.146447 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.146415 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") pod \"3c7f2418-d595-47ab-8e5d-1373653804fd\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " Sep 16 19:26:41.146625 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.146483 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle\") pod \"3c7f2418-d595-47ab-8e5d-1373653804fd\" (UID: \"3c7f2418-d595-47ab-8e5d-1373653804fd\") " Sep 16 19:26:41.146894 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.146860 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3c7f2418-d595-47ab-8e5d-1373653804fd" (UID: "3c7f2418-d595-47ab-8e5d-1373653804fd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:26:41.148532 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.148505 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c7f2418-d595-47ab-8e5d-1373653804fd" (UID: "3c7f2418-d595-47ab-8e5d-1373653804fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:26:41.248090 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.248023 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c7f2418-d595-47ab-8e5d-1373653804fd-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:26:41.248090 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.248045 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c7f2418-d595-47ab-8e5d-1373653804fd-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:26:41.886953 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.886917 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerID="bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790" exitCode=0 Sep 16 19:26:41.887233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.886966 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" Sep 16 19:26:41.887233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.887005 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" event={"ID":"3c7f2418-d595-47ab-8e5d-1373653804fd","Type":"ContainerDied","Data":"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790"} Sep 16 19:26:41.887233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.887061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4" event={"ID":"3c7f2418-d595-47ab-8e5d-1373653804fd","Type":"ContainerDied","Data":"1828a69304a32a55b21cd5d0f1d765e22d78b95af058ee8bdd05e201d587a64b"} Sep 16 19:26:41.887233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.887083 2559 scope.go:117] "RemoveContainer" containerID="bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790" Sep 16 19:26:41.895837 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.895818 2559 scope.go:117] "RemoveContainer" containerID="bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790" Sep 16 19:26:41.896087 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:26:41.896068 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790\": container with ID starting with bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790 not found: ID does not exist" containerID="bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790" Sep 16 19:26:41.896141 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.896095 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790"} err="failed to get container status \"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790\": rpc error: code = NotFound desc = could not find container \"bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790\": container with ID starting with bf16cf2096b756a6b511571ab264b9a8e27e0f221f9b6f99787b6e4000957790 not found: ID does not exist" Sep 16 19:26:41.903648 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.903626 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:26:41.907649 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:41.907628 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4a90a-5d7c796cdb-xsjm4"] Sep 16 19:26:43.354016 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:26:43.353976 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" path="/var/lib/kubelet/pods/3c7f2418-d595-47ab-8e5d-1373653804fd/volumes" Sep 16 19:29:31.515372 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:29:31.515253 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:29:31.520249 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:29:31.520223 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:33:45.204252 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:45.204218 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:33:45.206676 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:45.204432 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" containerID="cri-o://302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b" gracePeriod=30 Sep 16 19:33:48.359767 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:48.359722 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:33:53.359174 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:53.359135 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:33:58.359727 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:58.359684 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:33:58.360240 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:33:58.359822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:34:03.358992 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:03.358955 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:34:04.098708 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:04.098680 2559 ???:1] "http: TLS handshake error from 10.0.130.26:44272: EOF" Sep 16 19:34:04.101965 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:04.101943 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:04.921108 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:04.921054 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:05.733602 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:05.733571 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:06.516946 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:06.516905 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:07.344964 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:07.344931 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:08.140133 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:08.140103 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:08.359197 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:08.359162 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:34:08.907463 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:08.907431 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:09.696336 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:09.696296 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:10.500012 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:10.499979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:11.311362 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:11.311333 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:12.156822 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:12.156783 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:13.234455 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:13.234429 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:13.359226 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:13.359189 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:34:14.160445 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:14.160415 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:15.059385 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.059351 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-ca22e-64559f96c5-kscpd_359602d5-6cff-45e4-9e3d-f500149e3eab/switch-graph-ca22e/0.log" Sep 16 19:34:15.361525 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.361502 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:34:15.425819 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.425784 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") pod \"359602d5-6cff-45e4-9e3d-f500149e3eab\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " Sep 16 19:34:15.425971 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.425880 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle\") pod \"359602d5-6cff-45e4-9e3d-f500149e3eab\" (UID: \"359602d5-6cff-45e4-9e3d-f500149e3eab\") " Sep 16 19:34:15.426201 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.426177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "359602d5-6cff-45e4-9e3d-f500149e3eab" (UID: "359602d5-6cff-45e4-9e3d-f500149e3eab"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:34:15.427859 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.427825 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "359602d5-6cff-45e4-9e3d-f500149e3eab" (UID: "359602d5-6cff-45e4-9e3d-f500149e3eab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:34:15.526742 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.526707 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/359602d5-6cff-45e4-9e3d-f500149e3eab-proxy-tls\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:34:15.526742 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.526737 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359602d5-6cff-45e4-9e3d-f500149e3eab-openshift-service-ca-bundle\") on node \"ip-10-0-129-152.ec2.internal\" DevicePath \"\"" Sep 16 19:34:15.945087 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.945040 2559 generic.go:356] "Generic (PLEG): container finished" podID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerID="302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b" exitCode=0 Sep 16 19:34:15.945087 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.945085 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" Sep 16 19:34:15.945443 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.945094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" event={"ID":"359602d5-6cff-45e4-9e3d-f500149e3eab","Type":"ContainerDied","Data":"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b"} Sep 16 19:34:15.945443 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.945147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd" event={"ID":"359602d5-6cff-45e4-9e3d-f500149e3eab","Type":"ContainerDied","Data":"1079738d9065e681b0660d30167360b24654d1ab087f247454295b9ae11a40dc"} Sep 16 19:34:15.945443 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.945179 2559 scope.go:117] "RemoveContainer" containerID="302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b" Sep 16 19:34:15.954232 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.954209 2559 scope.go:117] "RemoveContainer" containerID="302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b" Sep 16 19:34:15.954550 ip-10-0-129-152 kubenswrapper[2559]: E0916 19:34:15.954530 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b\": container with ID starting with 302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b not found: ID does not exist" containerID="302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b" Sep 16 19:34:15.954630 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.954565 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b"} err="failed to get container status \"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b\": rpc error: code = NotFound desc = could not find container \"302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b\": container with ID starting with 302ce37e7b5f12fc7eba8cb2df750f9bd82a63d86ffbae4153dba748f694fb3b not found: ID does not exist" Sep 16 19:34:15.970098 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.970070 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:34:15.976454 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:15.976429 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ca22e-64559f96c5-kscpd"] Sep 16 19:34:17.353247 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:17.353218 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" path="/var/lib/kubelet/pods/359602d5-6cff-45e4-9e3d-f500149e3eab/volumes" Sep 16 19:34:18.271838 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.271801 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lpmdv/must-gather-wfbhn"] Sep 16 19:34:18.272243 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272225 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" Sep 16 19:34:18.272348 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272246 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" Sep 16 19:34:18.272348 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272266 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" Sep 16 19:34:18.272348 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272274 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" Sep 16 19:34:18.272522 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272401 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="359602d5-6cff-45e4-9e3d-f500149e3eab" containerName="switch-graph-ca22e" Sep 16 19:34:18.272522 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.272424 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c7f2418-d595-47ab-8e5d-1373653804fd" containerName="splitter-graph-4a90a" Sep 16 19:34:18.276933 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.276912 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.280233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.280213 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpmdv\"/\"kube-root-ca.crt\"" Sep 16 19:34:18.281098 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.281078 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpmdv\"/\"openshift-service-ca.crt\"" Sep 16 19:34:18.281194 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.281083 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lpmdv\"/\"default-dockercfg-87trx\"" Sep 16 19:34:18.287684 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.287661 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/must-gather-wfbhn"] Sep 16 19:34:18.350506 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.350479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92553e18-27c2-4ba5-9dd6-5629164d418f-must-gather-output\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.350615 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.350522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ct8j\" (UniqueName: \"kubernetes.io/projected/92553e18-27c2-4ba5-9dd6-5629164d418f-kube-api-access-8ct8j\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.451500 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.451469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92553e18-27c2-4ba5-9dd6-5629164d418f-must-gather-output\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.451831 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.451517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ct8j\" (UniqueName: \"kubernetes.io/projected/92553e18-27c2-4ba5-9dd6-5629164d418f-kube-api-access-8ct8j\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.451831 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.451782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92553e18-27c2-4ba5-9dd6-5629164d418f-must-gather-output\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.460982 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.460963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ct8j\" (UniqueName: \"kubernetes.io/projected/92553e18-27c2-4ba5-9dd6-5629164d418f-kube-api-access-8ct8j\") pod \"must-gather-wfbhn\" (UID: \"92553e18-27c2-4ba5-9dd6-5629164d418f\") " pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.603334 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.603292 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" Sep 16 19:34:18.871913 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.871890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/must-gather-wfbhn"] Sep 16 19:34:18.873942 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:34:18.873910 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92553e18_27c2_4ba5_9dd6_5629164d418f.slice/crio-b3bc73a1185503406a9b2eff0b1bbec1b539d5f3099fc365b318febfe800f003 WatchSource:0}: Error finding container b3bc73a1185503406a9b2eff0b1bbec1b539d5f3099fc365b318febfe800f003: Status 404 returned error can't find the container with id b3bc73a1185503406a9b2eff0b1bbec1b539d5f3099fc365b318febfe800f003 Sep 16 19:34:18.875812 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.875791 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:34:18.963284 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:18.963250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" event={"ID":"92553e18-27c2-4ba5-9dd6-5629164d418f","Type":"ContainerStarted","Data":"b3bc73a1185503406a9b2eff0b1bbec1b539d5f3099fc365b318febfe800f003"} Sep 16 19:34:19.971216 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:19.971180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" event={"ID":"92553e18-27c2-4ba5-9dd6-5629164d418f","Type":"ContainerStarted","Data":"af4c70c1e190e58a6e6868b2c8bb3a8a52b76b3f1fe63ca288618989e8cd431c"} Sep 16 19:34:19.971599 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:19.971228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" event={"ID":"92553e18-27c2-4ba5-9dd6-5629164d418f","Type":"ContainerStarted","Data":"ad2a7b6293173228fd9bf06d02c71f123314b3bf75c80d35eb79e9efcb14438e"} Sep 16 19:34:19.990079 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:19.990016 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lpmdv/must-gather-wfbhn" podStartSLOduration=1.120978276 podStartE2EDuration="1.98999682s" totalImagesPullingTime="869.018544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:19.98814042 +0000 UTC m=+2989.153160763" watchObservedRunningTime="2026-09-16 19:34:19.98999682 +0000 UTC m=+2989.155017163" Sep 16 19:34:21.222421 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:21.222392 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v9fdf_84b69c03-718b-41c6-be39-cbc758435e44/global-pull-secret-syncer/0.log" Sep 16 19:34:21.299099 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:21.299063 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-prgkg_9d498543-f479-431b-b239-f609a6704a22/konnectivity-agent/0.log" Sep 16 19:34:21.365545 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:21.365515 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-152.ec2.internal_5b66b66cfebc8e98c7f82d9151ac864b/haproxy/0.log" Sep 16 19:34:25.258990 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.258957 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/alertmanager/0.log" Sep 16 19:34:25.292390 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.292337 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/config-reloader/0.log" Sep 16 19:34:25.325767 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.325691 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/kube-rbac-proxy-web/0.log" Sep 16 19:34:25.372553 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.372520 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/kube-rbac-proxy/0.log" Sep 16 19:34:25.444334 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.444259 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/kube-rbac-proxy-metric/0.log" Sep 16 19:34:25.495862 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.495835 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/prom-label-proxy/0.log" Sep 16 19:34:25.530902 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.530828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_552cd12c-1573-4e04-81a7-a616831e3286/init-config-reloader/0.log" Sep 16 19:34:25.601928 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:25.601897 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-zh4wq_20a4c8a4-63bb-436b-bb44-a26b8f81a746/cluster-monitoring-operator/0.log" Sep 16 19:34:26.173147 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:26.173117 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9kd2_3b295457-aa8a-4e18-bad5-8572057a242b/node-exporter/0.log" Sep 16 19:34:26.215243 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:26.215202 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9kd2_3b295457-aa8a-4e18-bad5-8572057a242b/kube-rbac-proxy/0.log" Sep 16 19:34:26.243837 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:26.243803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9kd2_3b295457-aa8a-4e18-bad5-8572057a242b/init-textfile/0.log" Sep 16 19:34:26.830681 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:26.830601 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9lxtn_f478dce9-8b45-4beb-85ba-0ecc9692cf67/prometheus-operator/0.log" Sep 16 19:34:26.924163 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:26.924135 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9lxtn_f478dce9-8b45-4beb-85ba-0ecc9692cf67/kube-rbac-proxy/0.log" Sep 16 19:34:27.001434 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.001410 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-dn7gq_eae3fe52-bd69-4c4a-94b9-3900483187d8/prometheus-operator-admission-webhook/0.log" Sep 16 19:34:27.103740 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.103575 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6459b6f77b-gkmqx_6b16c5b7-27e7-43e9-906d-e1e9c5996f26/telemeter-client/0.log" Sep 16 19:34:27.159518 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.159488 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6459b6f77b-gkmqx_6b16c5b7-27e7-43e9-906d-e1e9c5996f26/reload/0.log" Sep 16 19:34:27.207417 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.207254 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6459b6f77b-gkmqx_6b16c5b7-27e7-43e9-906d-e1e9c5996f26/kube-rbac-proxy/0.log" Sep 16 19:34:27.266677 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.266650 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/thanos-query/0.log" Sep 16 19:34:27.300218 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.300189 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/kube-rbac-proxy-web/0.log" Sep 16 19:34:27.328583 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.328538 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/kube-rbac-proxy/0.log" Sep 16 19:34:27.363437 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.363348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/prom-label-proxy/0.log" Sep 16 19:34:27.393851 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.393819 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/kube-rbac-proxy-rules/0.log" Sep 16 19:34:27.429135 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:27.429109 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57496669bc-t98jp_f7573046-c4f6-4bc0-818c-1a38bbfd03a8/kube-rbac-proxy-metrics/0.log" Sep 16 19:34:28.489899 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:28.489826 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-rkvs9_fa283c3d-3f28-4add-9c56-59cf6c6c6bfc/networking-console-plugin/0.log" Sep 16 19:34:28.920552 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:28.920526 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:34:28.925969 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:28.925946 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/2.log" Sep 16 19:34:29.206421 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.206338 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw"] Sep 16 19:34:29.212037 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.212013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.224717 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.224689 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw"] Sep 16 19:34:29.363983 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.363949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-sys\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.364145 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.363995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc99t\" (UniqueName: \"kubernetes.io/projected/3f0e056c-9794-4e21-8bc0-9a0c812eae11-kube-api-access-lc99t\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.364145 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.364023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-lib-modules\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.364145 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.364082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-proc\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.364145 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.364133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-podres\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.370197 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.370174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8ff9558bd-s2ffn_3f01324e-4b51-43af-a1d3-96754e46c44d/console/0.log" Sep 16 19:34:29.407257 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.407233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-t6mt7_362e2e82-5781-437b-bc02-916e42f9e89f/download-server/0.log" Sep 16 19:34:29.465127 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.465033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-sys\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.465497 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.465478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc99t\" (UniqueName: \"kubernetes.io/projected/3f0e056c-9794-4e21-8bc0-9a0c812eae11-kube-api-access-lc99t\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466005 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.465974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-lib-modules\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466094 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.466048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-proc\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466136 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.466109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-podres\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466349 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.466315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-podres\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466399 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.465400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-sys\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466458 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.466444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-lib-modules\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.466508 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.466495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f0e056c-9794-4e21-8bc0-9a0c812eae11-proc\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.476669 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.476638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc99t\" (UniqueName: \"kubernetes.io/projected/3f0e056c-9794-4e21-8bc0-9a0c812eae11-kube-api-access-lc99t\") pod \"perf-node-gather-daemonset-76kmw\" (UID: \"3f0e056c-9794-4e21-8bc0-9a0c812eae11\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.525894 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.525662 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:29.837591 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:29.837511 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw"] Sep 16 19:34:29.843569 ip-10-0-129-152 kubenswrapper[2559]: W0916 19:34:29.843540 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f0e056c_9794_4e21_8bc0_9a0c812eae11.slice/crio-351abbd8b565f26ad8e4be46fe07584603e0457098c7d36a2f87a59f8b40bb7a WatchSource:0}: Error finding container 351abbd8b565f26ad8e4be46fe07584603e0457098c7d36a2f87a59f8b40bb7a: Status 404 returned error can't find the container with id 351abbd8b565f26ad8e4be46fe07584603e0457098c7d36a2f87a59f8b40bb7a Sep 16 19:34:30.042852 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:30.042817 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" event={"ID":"3f0e056c-9794-4e21-8bc0-9a0c812eae11","Type":"ContainerStarted","Data":"dc3f7c2f8a45cd773bd5efd2fbece0930e399edd98bcc87bf1fd92a482375c2b"} Sep 16 19:34:30.043114 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:30.043094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" event={"ID":"3f0e056c-9794-4e21-8bc0-9a0c812eae11","Type":"ContainerStarted","Data":"351abbd8b565f26ad8e4be46fe07584603e0457098c7d36a2f87a59f8b40bb7a"} Sep 16 19:34:30.043345 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:30.043268 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:30.064481 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:30.064425 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" podStartSLOduration=1.064406551 podStartE2EDuration="1.064406551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:30.061924422 +0000 UTC m=+2999.226944764" watchObservedRunningTime="2026-09-16 19:34:30.064406551 +0000 UTC m=+2999.229426896" Sep 16 19:34:31.040969 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.040939 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w22ll_772f55b7-eecf-4c9a-93a6-547a73a0b147/dns/0.log" Sep 16 19:34:31.065979 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.065907 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w22ll_772f55b7-eecf-4c9a-93a6-547a73a0b147/kube-rbac-proxy/0.log" Sep 16 19:34:31.096623 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.096595 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2qwhj_14e13b94-9f75-43d9-a98b-8899a3926385/dns-node-resolver/0.log" Sep 16 19:34:31.554170 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.554135 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:34:31.559150 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.559125 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ts4c_f2064f60-12eb-4766-bf70-d91ed2f5845c/console-operator/1.log" Sep 16 19:34:31.626695 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.626669 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-959db6b9-vbg5r_68cae0d6-243d-4b85-8877-9b6f38ef3771/registry/0.log" Sep 16 19:34:31.721733 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:31.721701 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rk528_c624e892-e030-4e19-a78c-c9a24b6abc1b/node-ca/0.log" Sep 16 19:34:32.918664 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:32.918636 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k9wjj_a470b680-0106-40e5-aab1-71f7f33a0598/serve-healthcheck-canary/0.log" Sep 16 19:34:33.671977 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:33.671948 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7fvd_16f87351-5974-49af-86f4-b91e72da6b85/kube-rbac-proxy/0.log" Sep 16 19:34:33.711607 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:33.711579 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7fvd_16f87351-5974-49af-86f4-b91e72da6b85/exporter/0.log" Sep 16 19:34:33.772725 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:33.772697 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r7fvd_16f87351-5974-49af-86f4-b91e72da6b85/extractor/0.log" Sep 16 19:34:36.059381 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:36.059352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-76kmw" Sep 16 19:34:36.232656 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:36.232621 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-jmz9p_60686baf-570b-416f-85ea-7e620b9914d8/manager/0.log" Sep 16 19:34:36.272576 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:36.272550 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dxlzc_657e282d-6828-409e-b268-a29e4815f51d/server/0.log" Sep 16 19:34:41.017863 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:41.017832 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kh8ps_f3345352-c4b7-4b44-9372-3e5c21275e17/migrator/0.log" Sep 16 19:34:41.046191 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:41.046167 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kh8ps_f3345352-c4b7-4b44-9372-3e5c21275e17/graceful-termination/0.log" Sep 16 19:34:41.393007 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:41.392973 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lx4jp_ae82780d-8d4b-43d4-a9f7-a2926d86ba1b/kube-storage-version-migrator-operator/1.log" Sep 16 19:34:41.393750 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:41.393726 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lx4jp_ae82780d-8d4b-43d4-a9f7-a2926d86ba1b/kube-storage-version-migrator-operator/0.log" Sep 16 19:34:42.514719 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.514688 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/kube-multus-additional-cni-plugins/0.log" Sep 16 19:34:42.537707 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.537683 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/egress-router-binary-copy/0.log" Sep 16 19:34:42.564812 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.564774 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/cni-plugins/0.log" Sep 16 19:34:42.587531 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.587504 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/bond-cni-plugin/0.log" Sep 16 19:34:42.610027 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.610002 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/routeoverride-cni/0.log" Sep 16 19:34:42.631801 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.631777 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/whereabouts-cni-bincopy/0.log" Sep 16 19:34:42.656490 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.656459 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n2khs_75eea085-6c20-456f-b234-ef793183e012/whereabouts-cni/0.log" Sep 16 19:34:42.896378 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:42.896352 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lk2j8_af372705-98be-4c8f-8cdb-c56f8735b57f/kube-multus/0.log" Sep 16 19:34:43.076421 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:43.076395 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7qvp_0c6db619-fa2b-4eae-9e3e-1b94fc47bba3/network-metrics-daemon/0.log" Sep 16 19:34:43.099786 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:43.099742 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7qvp_0c6db619-fa2b-4eae-9e3e-1b94fc47bba3/kube-rbac-proxy/0.log" Sep 16 19:34:44.100551 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.100520 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/ovn-controller/0.log" Sep 16 19:34:44.119918 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.119892 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/ovn-acl-logging/0.log" Sep 16 19:34:44.143759 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.143738 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/kube-rbac-proxy-node/0.log" Sep 16 19:34:44.164936 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.164912 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:34:44.185233 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.185207 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/northd/0.log" Sep 16 19:34:44.206965 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.206942 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/nbdb/0.log" Sep 16 19:34:44.228966 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.228946 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/sbdb/0.log" Sep 16 19:34:44.269853 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:44.269828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mhx6h_b5826602-57d8-4dc5-85e7-40ff7790798a/ovnkube-controller/0.log" Sep 16 19:34:45.415074 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:45.415050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-96jtp_fa6f9a4a-929c-4bd0-9df5-7a331970122d/check-endpoints/0.log" Sep 16 19:34:45.462384 ip-10-0-129-152 kubenswrapper[2559]: I0916 19:34:45.462355 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vpt6j_3b9dd24b-6c17-4242-9237-c9a757140681/network-check-target-container/0.log"