Jul 07 04:23:46.901041 ip-10-0-142-140 systemd[1]: Starting Kubernetes Kubelet... Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.329614 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335280 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335293 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335297 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335300 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335304 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335307 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335309 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.370457 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335312 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335315 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335318 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335321 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335324 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335326 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335331 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335334 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335338 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335341 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335343 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335346 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335355 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335358 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335361 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335364 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335368 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335372 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335375 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.374051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335379 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335382 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335384 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335387 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335390 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335393 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335395 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335405 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335407 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335410 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335413 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335415 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335418 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335420 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335423 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335426 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335428 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335432 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335435 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.387167 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335437 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335440 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335442 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335445 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335448 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335450 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335453 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335455 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335458 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335462 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335464 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335467 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335470 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335472 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335474 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335478 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335481 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335483 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335486 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335488 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.388441 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335491 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335493 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335496 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335498 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335501 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335504 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335506 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335509 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335511 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335514 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335517 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335520 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335523 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335525 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335528 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335530 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335532 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335535 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335538 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335540 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.406551 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335542 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335546 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335550 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335553 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335556 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335558 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335561 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335563 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335566 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335569 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335571 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335573 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335576 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.335578 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336273 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336281 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336283 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336288 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336291 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.410997 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336294 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.408115 ip-10-0-142-140 systemd[1]: Started Kubernetes Kubelet. Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336297 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336299 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336302 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336305 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336308 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336311 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336313 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336316 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336319 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336321 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336324 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336327 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336329 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336332 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336334 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336337 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336340 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336342 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336345 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.423705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336348 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336350 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336353 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336355 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336359 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336361 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336364 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336367 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336369 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336372 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336375 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336377 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336379 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336382 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336384 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336387 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336390 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336392 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336395 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336398 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.627633 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336400 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336403 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336406 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336408 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336411 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336413 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336415 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336418 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336431 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336435 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336439 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336442 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336445 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336447 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336450 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336452 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336455 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336458 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336460 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.646214 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336464 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336468 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336470 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336474 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336477 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336479 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336483 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336486 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336488 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336491 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336494 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336496 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336500 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336503 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336505 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336508 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336510 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336513 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336515 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336517 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.651533 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336520 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336522 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336526 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336528 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336530 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336533 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336539 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336541 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336544 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336546 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336549 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336551 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336555 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336558 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336561 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336974 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336980 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336983 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336985 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.652248 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336988 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336991 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336994 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.336997 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337000 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337003 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337006 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337008 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337010 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337013 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337015 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337018 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337021 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337023 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337025 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337028 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337032 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337036 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337038 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.652829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337043 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337046 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337048 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337051 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337053 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337056 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337059 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337062 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337064 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337067 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337069 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337072 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337075 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337077 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337080 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337083 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337085 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337088 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337092 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.653469 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337095 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337098 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337101 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337104 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337106 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337109 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337112 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337114 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337118 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337122 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337125 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337127 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337130 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337134 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337137 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337141 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337143 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337146 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337149 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.654051 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337151 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337153 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337156 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337159 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337161 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337164 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337166 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337169 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337171 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337174 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337176 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337179 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337181 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337184 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337191 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337193 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337196 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337198 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337201 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337203 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.654660 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337206 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337208 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337211 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337213 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337215 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337218 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337222 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337224 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337228 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337230 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337233 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337236 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337239 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337242 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337244 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337247 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337249 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337252 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337314 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337321 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337326 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 04:23:47.655388 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337331 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337334 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337338 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337342 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337345 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337349 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337352 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337356 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337359 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337362 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337365 2563 flags.go:64] FLAG: --cgroup-root="" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337367 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337370 2563 flags.go:64] FLAG: --client-ca-file="" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337373 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337376 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337380 2563 flags.go:64] FLAG: --cluster-domain="" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337390 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337397 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337400 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337404 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337408 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337411 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337414 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 04:23:47.656143 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337417 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337420 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337423 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337426 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337429 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337432 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337436 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337439 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337442 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337444 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337447 2563 flags.go:64] FLAG: --enable-server="true" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337450 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337454 2563 flags.go:64] FLAG: --event-burst="100" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337458 2563 flags.go:64] FLAG: --event-qps="50" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337460 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337463 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337467 2563 flags.go:64] FLAG: --eviction-hard="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337475 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337478 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337481 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337484 2563 flags.go:64] FLAG: --eviction-soft="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337486 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337489 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337492 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337495 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 04:23:47.656818 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337498 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337502 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337505 2563 flags.go:64] FLAG: --feature-gates="" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337509 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337512 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337515 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337518 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337521 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337525 2563 flags.go:64] FLAG: --help="false" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337528 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337531 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337533 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337537 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337540 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337543 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337546 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337548 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337551 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337554 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337557 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337560 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337562 2563 flags.go:64] FLAG: --kube-reserved="" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337565 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337568 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 04:23:47.657582 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337571 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337574 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337577 2563 flags.go:64] FLAG: --lock-file="" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337579 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337582 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337585 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337590 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337593 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337597 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337603 2563 flags.go:64] FLAG: --logging-format="text" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337607 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337612 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337615 2563 flags.go:64] FLAG: --manifest-url="" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337618 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337622 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337625 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337629 2563 flags.go:64] FLAG: --max-pods="110" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337632 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337634 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337637 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337640 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337643 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337646 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337649 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337656 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 04:23:47.658223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337659 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337662 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337665 2563 flags.go:64] FLAG: --pod-cidr="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337667 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337674 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337676 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337679 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337682 2563 flags.go:64] FLAG: --port="10250" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337689 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337692 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b91cf243068d3422" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337695 2563 flags.go:64] FLAG: --qos-reserved="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337698 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337701 2563 flags.go:64] FLAG: --register-node="true" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337703 2563 flags.go:64] FLAG: --register-with-taints="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337707 2563 flags.go:64] FLAG: --registry-burst="10" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337710 2563 flags.go:64] FLAG: --registry-qps="5" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337714 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337717 2563 flags.go:64] FLAG: --reserved-memory="" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337720 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337724 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337727 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337731 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337733 2563 flags.go:64] FLAG: --runonce="false" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337736 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337739 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 04:23:47.659078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337742 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337745 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337747 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337750 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337753 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337756 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337759 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337762 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337778 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337781 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337784 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337787 2563 flags.go:64] FLAG: --system-cgroups="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337790 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337795 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337798 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337801 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337806 2563 flags.go:64] FLAG: --tls-min-version="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337809 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337812 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337814 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337817 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337820 2563 flags.go:64] FLAG: --v="2" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337823 2563 flags.go:64] FLAG: --version="false" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337828 2563 flags.go:64] FLAG: --vmodule="" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337832 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 04:23:47.660244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.337835 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337947 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337951 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337953 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337956 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337959 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337962 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337964 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337967 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337969 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337972 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337974 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337977 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337979 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337982 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337984 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337988 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337992 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337994 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.337997 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.662338 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338000 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338003 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338006 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338008 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338011 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338014 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338017 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338020 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338023 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338026 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338030 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338032 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338035 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338038 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338041 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338044 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338046 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338049 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338053 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.663400 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338056 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338059 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338062 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338065 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338068 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338070 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338073 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338075 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338078 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338080 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338083 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338085 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338088 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338090 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338093 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338095 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338098 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338100 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338104 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338106 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.664433 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338109 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338111 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338114 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338117 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338120 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338122 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338126 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338129 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338131 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338133 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338136 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338138 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338141 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338143 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338146 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338148 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338151 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338153 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338156 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338158 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.665625 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338161 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338163 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338166 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338168 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338171 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338173 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338176 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338178 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338180 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338183 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338185 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338188 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338191 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338193 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338196 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338201 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338203 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338206 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338209 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338212 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.666814 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.338214 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.338911 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.346915 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.346931 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.346947 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.346953 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347015 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347020 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347023 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347026 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347029 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347032 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347034 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347037 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347040 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.667603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347042 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347045 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347048 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347050 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347053 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347057 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347060 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347063 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347066 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347068 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347071 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347074 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347076 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347079 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347082 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347085 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347087 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347090 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347093 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347096 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.668221 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347099 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347101 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347104 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347107 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347109 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347112 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347115 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347117 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347120 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347123 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347125 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347128 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347130 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347133 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347135 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347138 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347146 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347150 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347153 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.819584 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347156 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347159 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347163 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347166 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347168 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347171 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347175 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347177 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347180 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347182 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347185 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347188 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347190 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347192 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347195 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347197 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347200 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347202 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347205 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347208 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.820213 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347210 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347213 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347216 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347220 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347222 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347225 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347227 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347230 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347233 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347236 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347239 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347242 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347245 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347248 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347250 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347253 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347255 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347258 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347261 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.824112 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347263 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347266 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347270 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347273 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347276 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347279 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347282 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347284 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347286 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347289 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347292 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347294 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.347299 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347420 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347425 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:23:47.824647 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347428 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347431 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347434 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347436 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347439 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347442 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347445 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347447 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347451 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347454 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347457 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347460 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347462 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347465 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347468 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347471 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347473 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347476 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347478 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347481 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:23:47.825076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347483 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347486 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347489 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347491 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347494 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347496 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347499 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347501 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347504 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347506 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347509 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347512 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347514 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347518 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347521 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347523 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347526 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347529 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347531 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347535 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:23:47.825631 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347539 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347542 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347545 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347549 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347552 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347554 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347557 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347560 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347563 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347566 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347568 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347571 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347574 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347576 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347579 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347581 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347584 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347587 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347589 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:23:47.826182 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347592 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347594 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347597 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347599 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347602 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347605 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347608 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347611 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347614 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347616 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347619 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347621 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347624 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347628 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347630 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347632 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347635 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347638 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347641 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347643 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:23:47.826722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347646 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347648 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347651 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347653 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347656 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347658 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347661 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347665 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347668 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347670 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347673 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347675 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347678 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347680 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347683 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347686 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347688 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:23:47.827314 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:47.347692 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.347698 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.347841 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.351400 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.353017 2563 server.go:1019] "Starting client certificate rotation" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.353116 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.353155 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.355752 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.357051 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.368590 2563 log.go:25] "Validated CRI v1 runtime API" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.371971 2563 log.go:25] "Validated CRI v1 image API" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.374762 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 04:23:47.931192 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.377667 2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 863aef71-58c3-409e-a00a-732fe972ad0d:/dev/nvme0n1p4 d7b1f795-5ca5-40ea-aa09-9981aa36efcd:/dev/nvme0n1p3] Jul 07 04:23:47.931568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.377685 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 04:23:47.931568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.384108 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.384232 2563 manager.go:216] Machine: {Timestamp:2026-07-07 04:23:47.382886797 +0000 UTC m=+0.364820643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099618 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27234a96e8f02c116e03f0b855a421 SystemUUID:ec27234a-96e8-f02c-116e-03f0b855a421 BootID:c8dd8de5-51aa-4a63-8bcf-fc9645864710 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:33:18:bf:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:33:18:bf:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:e0:2e:98:c7:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.384329 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.384404 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.387740 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.387780 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.389412 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.389426 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.389435 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.389449 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.391320 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396021 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396115 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396141 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396150 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396167 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396179 2563 kubelet.go:398] "Adding apiserver pod source" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396187 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 04:23:47.931978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396449 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.396462 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.398727 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.401217 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.401244 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402697 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402712 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402718 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402723 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402732 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402738 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402752 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402758 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402778 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402785 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402797 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.402803 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.403709 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.403716 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.407133 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.407155 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.407297 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.407496 2563 server.go:1268] "Started kubelet" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.407622 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.407614 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.407676 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.408999 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.411396 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.413943 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-140.ec2.internal.18bfe73d2daca0be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-140.ec2.internal,UID:ip-10-0-142-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-140.ec2.internal,},FirstTimestamp:2026-07-07 04:23:47.407315134 +0000 UTC m=+0.389248976,LastTimestamp:2026-07-07 04:23:47.407315134 +0000 UTC m=+0.389248976,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-140.ec2.internal,}" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.417018 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.419698 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.420199 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.420758 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421192 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421193 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421230 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421398 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421408 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421500 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421518 2563 factory.go:55] Registering systemd factory Jul 07 04:23:47.932596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421527 2563 factory.go:223] Registration of the systemd container factory successfully Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421737 2563 factory.go:153] Registering CRI-O factory Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421751 2563 factory.go:223] Registration of the crio container factory successfully Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421785 2563 factory.go:103] Registering Raw factory Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.421797 2563 manager.go:1196] Started watching for new ooms in manager Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.422148 2563 manager.go:349] Starting recovery of all containers Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.422365 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.422452 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.430995 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccvms" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.433414 2563 manager.go:354] Recovery completed Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.436730 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccvms" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.437905 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.439584 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.439608 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.439621 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440134 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440143 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440158 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440195 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440923 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440955 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.440991 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.442139 2563 policy_none.go:49] "None policy: Start" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.442152 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.442162 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.444160 2563 policy_none.go:47] "Start" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506324 2563 manager.go:337] "Starting Device Plugin manager" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.506371 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506385 2563 server.go:87] "Starting device plugin registration server" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506638 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506651 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506742 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506855 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.506864 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.507228 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.507272 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.519924 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.521089 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.521121 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.521144 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.521182 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.523407 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.607362 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.608323 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.608347 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.608359 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.608380 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.934035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.618636 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.618654 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-140.ec2.internal\": node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.621210 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"] Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.621265 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.621568 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.621591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.622755 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.622797 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.622807 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.625080 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.625171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.625197 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626005 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626013 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626035 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626044 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626048 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.626059 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.628261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.628289 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.629065 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.629091 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.629102 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.646849 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.661325 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-140.ec2.internal\" not found" node="ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.667220 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-140.ec2.internal\" not found" node="ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.722082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.722153 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.935278 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.722172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.722227 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f00651057cf051b0e8c326f620be66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"5f00651057cf051b0e8c326f620be66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.747598 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.822992 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44f23d15227f2b810bc3ab580d4a5aaa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"44f23d15227f2b810bc3ab580d4a5aaa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.848439 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.924079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44f23d15227f2b810bc3ab580d4a5aaa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"44f23d15227f2b810bc3ab580d4a5aaa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:47.936095 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.924125 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44f23d15227f2b810bc3ab580d4a5aaa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"44f23d15227f2b810bc3ab580d4a5aaa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:48.061257 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:47.949192 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.061257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.964327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:48.061257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:47.970267 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:48.061257 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:48.049922 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.166307 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:48.150451 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.289900 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:48.250856 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.289900 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.274739 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:48.433037 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.351814 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 04:23:48.433037 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:48.351824 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.433037 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.351953 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 04:23:48.433037 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.351954 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 04:23:48.544454 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.442013 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 04:18:48 +0000 UTC" deadline="2028-02-22 09:32:51.188847923 +0000 UTC" Jul 07 04:23:48.544454 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.442037 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14285h9m2.74681332s" Jul 07 04:23:48.544454 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:48.452946 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found" Jul 07 04:23:48.631054 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.550102 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:48.631054 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:48.618602 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f00651057cf051b0e8c326f620be66f.slice/crio-50d64407a864e030724a33e2a6bd2135bcaf1e2b636bef62805d9fc3aa256afa WatchSource:0}: Error finding container 50d64407a864e030724a33e2a6bd2135bcaf1e2b636bef62805d9fc3aa256afa: Status 404 returned error can't find the container with id 50d64407a864e030724a33e2a6bd2135bcaf1e2b636bef62805d9fc3aa256afa Jul 07 04:23:48.631054 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:48.618937 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f23d15227f2b810bc3ab580d4a5aaa.slice/crio-029841b59ace50fe2872c400c22d2be298a695d5d51f01724c7e70bfe189b1bf WatchSource:0}: Error finding container 029841b59ace50fe2872c400c22d2be298a695d5d51f01724c7e70bfe189b1bf: Status 404 returned error can't find the container with id 029841b59ace50fe2872c400c22d2be298a695d5d51f01724c7e70bfe189b1bf Jul 07 04:23:48.631054 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.621120 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" Jul 07 04:23:48.631054 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.623318 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:23:48.784398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.634661 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 04:23:48.784398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.635639 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" Jul 07 04:23:48.784398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.644140 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 04:23:48.784398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:48.774135 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.397451 2563 apiserver.go:52] "Watching apiserver" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.405509 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.405932 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-h54j2","openshift-dns/node-resolver-gvmr2","openshift-image-registry/node-ca-c8tf5","openshift-multus/multus-additional-cni-plugins-z8fdh","openshift-multus/multus-l26zw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal","openshift-multus/network-metrics-daemon-7mhwl","openshift-network-diagnostics/network-check-target-bn4nk","openshift-network-operator/iptables-alerter-vdnwl","openshift-ovn-kubernetes/ovnkube-node-bhbbm","kube-system/global-pull-secret-syncer-2tvqx","kube-system/konnectivity-agent-mvhjd","kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"] Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.408958 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.411322 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.413465 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.416113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.418834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.418607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.421994 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.421164 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 04:23:49.421994 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.421878 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.425103 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.424096 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:49.425103 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.424170 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:49.428667 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.426809 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.428667 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.426893 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:49.431436 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.429652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.432839 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.435236 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.437797 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.437870 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgz4z\" (UniqueName: \"kubernetes.io/projected/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kube-api-access-rgz4z\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-cnibin\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439872 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-cni-binary-copy\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439903 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38057ab-6fb0-4a71-b834-79b76eaea8d8-iptables-alerter-script\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.439986 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42292ea1-1500-4541-9321-06324275ce72-serviceca\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440011 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5dfr\" (UniqueName: \"kubernetes.io/projected/d38057ab-6fb0-4a71-b834-79b76eaea8d8-kube-api-access-f5dfr\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-hostroot\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440063 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71128bfb-74e3-45f7-a7c1-eca531c3cad8-agent-certs\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440167 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-netns\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.491207 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-multus\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-multus-certs\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-device-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-system-cni-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71128bfb-74e3-45f7-a7c1-eca531c3cad8-konnectivity-ca\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440367 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-system-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-cnibin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440413 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-k8s-cni-cncf-io\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440439 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jfc\" (UniqueName: \"kubernetes.io/projected/c6af0dfb-4a51-4173-8611-6b091dd6751c-kube-api-access-45jfc\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-sys-fs\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42292ea1-1500-4541-9321-06324275ce72-host\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440507 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440556 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-socket-dir-parent\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-registration-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7d7b\" (UniqueName: \"kubernetes.io/projected/d8436039-fc85-4333-88f1-d8ab9671e473-kube-api-access-x7d7b\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.494463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38057ab-6fb0-4a71-b834-79b76eaea8d8-host-slash\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-os-release\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-bin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-kubelet\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-conf-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-daemon-config\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-os-release\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440959 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.440984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.441008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlmg\" (UniqueName: \"kubernetes.io/projected/42292ea1-1500-4541-9321-06324275ce72-kube-api-access-hjlmg\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.441060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-etc-kubernetes\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.441089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-socket-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.453104 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.453134 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.453287 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.495434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.453331 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.523145 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.537743 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"5f00651057cf051b0e8c326f620be66f","Type":"ContainerStarted","Data":"50d64407a864e030724a33e2a6bd2135bcaf1e2b636bef62805d9fc3aa256afa"} Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.540010 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" event={"ID":"44f23d15227f2b810bc3ab580d4a5aaa","Type":"ContainerStarted","Data":"029841b59ace50fe2872c400c22d2be298a695d5d51f01724c7e70bfe189b1bf"} Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-netns\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-multus\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-multus-certs\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542334 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-ovn\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-system-cni-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542405 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-netns\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542418 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-multus-certs\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542429 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71128bfb-74e3-45f7-a7c1-eca531c3cad8-konnectivity-ca\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542433 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-multus\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-system-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-cnibin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542687 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-cnibin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.562043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542687 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-system-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542744 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-system-cni-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542765 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-sys-fs\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-conf\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-host\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42292ea1-1500-4541-9321-06324275ce72-host\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542952 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-registration-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.542984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-etc-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-log-socket\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-bin\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543112 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-config\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65c98535-b722-4529-b0b6-efdd4138b9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543177 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7d7b\" (UniqueName: \"kubernetes.io/projected/d8436039-fc85-4333-88f1-d8ab9671e473-kube-api-access-x7d7b\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-os-release\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-kubelet\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.563089 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-etc-kubernetes\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-systemd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543325 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-env-overrides\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543363 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmfg\" (UniqueName: \"kubernetes.io/projected/137a0240-63ec-41ac-910d-aafdbcfc59f0-kube-api-access-kbmfg\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlmg\" (UniqueName: \"kubernetes.io/projected/42292ea1-1500-4541-9321-06324275ce72-kube-api-access-hjlmg\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgz4z\" (UniqueName: \"kubernetes.io/projected/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kube-api-access-rgz4z\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71128bfb-74e3-45f7-a7c1-eca531c3cad8-konnectivity-ca\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-systemd\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543519 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-sys-fs\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543598 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-kubelet\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543647 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-etc-kubernetes\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543889 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42292ea1-1500-4541-9321-06324275ce72-host\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.543945 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-registration-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544014 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-os-release\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544121 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-run\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544166 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-lib-modules\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-netd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.564398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-script-lib\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-cnibin\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544397 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-netns\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42292ea1-1500-4541-9321-06324275ce72-serviceca\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544474 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-kubelet-config\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5dfr\" (UniqueName: \"kubernetes.io/projected/d38057ab-6fb0-4a71-b834-79b76eaea8d8-kube-api-access-f5dfr\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-hostroot\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-modprobe-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-var-lib-kubelet\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544715 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71128bfb-74e3-45f7-a7c1-eca531c3cad8-agent-certs\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-device-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544868 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-k8s-cni-cncf-io\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.544989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jfc\" (UniqueName: \"kubernetes.io/projected/c6af0dfb-4a51-4173-8611-6b091dd6751c-kube-api-access-45jfc\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.565463 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545024 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-tuned\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-kubelet\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-systemd-units\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqwv\" (UniqueName: \"kubernetes.io/projected/65c98535-b722-4529-b0b6-efdd4138b9ae-kube-api-access-tfqwv\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-socket-dir-parent\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-sys\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-node-log\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr96\" (UniqueName: \"kubernetes.io/projected/e58e6ac7-8ee6-4790-8625-54d10183203e-kube-api-access-fmr96\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545364 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38057ab-6fb0-4a71-b834-79b76eaea8d8-host-slash\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-bin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-conf-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-daemon-config\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysconfig\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-hostroot\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545695 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-run-k8s-cni-cncf-io\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.566690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-os-release\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545918 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-cni-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545961 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-socket-dir-parent\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-socket-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546022 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-host-var-lib-cni-bin\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546037 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58e6ac7-8ee6-4790-8625-54d10183203e-tmp-dir\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546066 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38057ab-6fb0-4a71-b834-79b76eaea8d8-host-slash\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.545819 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546117 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-conf-dir\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546129 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-os-release\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-cni-binary-copy\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546253 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-socket-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546339 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8888cdf0-ef1e-485f-83e7-80ac2797e99c-device-dir\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.567738 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-tmp\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546468 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8436039-fc85-4333-88f1-d8ab9671e473-cnibin\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38057ab-6fb0-4a71-b834-79b76eaea8d8-iptables-alerter-script\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7thm\" (UniqueName: \"kubernetes.io/projected/ed3fbfbc-9a18-416a-bc05-f53bafee5058-kube-api-access-t7thm\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-slash\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546863 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-dbus\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.546960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-kubernetes\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.547575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-var-lib-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.547610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e58e6ac7-8ee6-4790-8625-54d10183203e-hosts-file\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563047 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4cpvg\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563069 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563380 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563448 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563675 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x9lg2\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563685 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563784 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lpgd\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.563889 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564079 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564164 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564169 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564303 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tt8j8\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564338 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564227 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-69shv\"" Jul 07 04:23:49.568957 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564449 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564571 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.564843 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.565018 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.565291 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.565326 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jmdfk\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.565682 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.565935 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnsjj\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.567058 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.567537 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38057ab-6fb0-4a71-b834-79b76eaea8d8-iptables-alerter-script\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.568320 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.568340 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.568652 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.568937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9m5k7\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.569002 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fjm6l\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.569114 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.569291 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.569351 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.569486 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.568938 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570045 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570127 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570215 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-multus-daemon-config\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570260 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6af0dfb-4a51-4173-8611-6b091dd6751c-cni-binary-copy\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.570652 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570351 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8436039-fc85-4333-88f1-d8ab9671e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.570728 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42292ea1-1500-4541-9321-06324275ce72-serviceca\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.571189 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71128bfb-74e3-45f7-a7c1-eca531c3cad8-agent-certs\") pod \"konnectivity-agent-mvhjd\" (UID: \"71128bfb-74e3-45f7-a7c1-eca531c3cad8\") " pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.572941 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.572962 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.572977 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.573053 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:50.073032798 +0000 UTC m=+3.054966635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.574579 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.577978 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlmg\" (UniqueName: \"kubernetes.io/projected/42292ea1-1500-4541-9321-06324275ce72-kube-api-access-hjlmg\") pod \"node-ca-c8tf5\" (UID: \"42292ea1-1500-4541-9321-06324275ce72\") " pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.578468 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5dfr\" (UniqueName: \"kubernetes.io/projected/d38057ab-6fb0-4a71-b834-79b76eaea8d8-kube-api-access-f5dfr\") pod \"iptables-alerter-vdnwl\" (UID: \"d38057ab-6fb0-4a71-b834-79b76eaea8d8\") " pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.580115 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgz4z\" (UniqueName: \"kubernetes.io/projected/8888cdf0-ef1e-485f-83e7-80ac2797e99c-kube-api-access-rgz4z\") pod \"aws-ebs-csi-driver-node-v749r\" (UID: \"8888cdf0-ef1e-485f-83e7-80ac2797e99c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.581532 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7d7b\" (UniqueName: \"kubernetes.io/projected/d8436039-fc85-4333-88f1-d8ab9671e473-kube-api-access-x7d7b\") pod \"multus-additional-cni-plugins-z8fdh\" (UID: \"d8436039-fc85-4333-88f1-d8ab9671e473\") " pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.584312 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jfc\" (UniqueName: \"kubernetes.io/projected/c6af0dfb-4a51-4173-8611-6b091dd6751c-kube-api-access-45jfc\") pod \"multus-l26zw\" (UID: \"c6af0dfb-4a51-4173-8611-6b091dd6751c\") " pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.602227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.600291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:23:49.609482 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.609465 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c8tf5" Jul 07 04:23:49.611017 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.610978 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gtw8f" Jul 07 04:23:49.619736 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.619393 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gtw8f" Jul 07 04:23:49.622194 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:49.622165 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71128bfb_74e3_45f7_a7c1_eca531c3cad8.slice/crio-8980a3db6bd9eb4266f44b8c681eab98eb067c24a3a5314b710b988188f3c2d5 WatchSource:0}: Error finding container 8980a3db6bd9eb4266f44b8c681eab98eb067c24a3a5314b710b988188f3c2d5: Status 404 returned error can't find the container with id 8980a3db6bd9eb4266f44b8c681eab98eb067c24a3a5314b710b988188f3c2d5 Jul 07 04:23:49.624441 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.624419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" Jul 07 04:23:49.626252 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:49.625896 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42292ea1_1500_4541_9321_06324275ce72.slice/crio-cb07aff70cbf425c7922e4e73b898c17437c1b6353d7c57c8cc801db2e720c5c WatchSource:0}: Error finding container cb07aff70cbf425c7922e4e73b898c17437c1b6353d7c57c8cc801db2e720c5c: Status 404 returned error can't find the container with id cb07aff70cbf425c7922e4e73b898c17437c1b6353d7c57c8cc801db2e720c5c Jul 07 04:23:49.632300 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.632145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l26zw" Jul 07 04:23:49.644459 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.644256 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" Jul 07 04:23:49.646003 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.645811 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.647905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-slash\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.647944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-dbus\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.647972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-kubernetes\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.647999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-var-lib-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e58e6ac7-8ee6-4790-8625-54d10183203e-hosts-file\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648081 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-ovn\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-conf\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648192 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-slash\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-host\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-etc-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648276 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-host\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648324 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-log-socket\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648341 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-dbus\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648377 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e58e6ac7-8ee6-4790-8625-54d10183203e-hosts-file\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648405 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-kubernetes\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648423 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-ovn\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648450 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-etc-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648467 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-log-socket\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648706 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-conf\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-bin\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648756 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-var-lib-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648756 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648823 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-bin\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-config\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65c98535-b722-4529-b0b6-efdd4138b9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-systemd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-env-overrides\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.648971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmfg\" (UniqueName: \"kubernetes.io/projected/137a0240-63ec-41ac-910d-aafdbcfc59f0-kube-api-access-kbmfg\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-systemd\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649073 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-systemd\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649487 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-env-overrides\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649588 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-config\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649639 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-run\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-lib-modules\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-netd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-script-lib\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-netns\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-kubelet-config\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649895 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-modprobe-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-var-lib-kubelet\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.649996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-tuned\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-kubelet\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-systemd-units\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650052 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-systemd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqwv\" (UniqueName: \"kubernetes.io/projected/65c98535-b722-4529-b0b6-efdd4138b9ae-kube-api-access-tfqwv\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-sys\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.650932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650103 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-kubelet-config\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-node-log\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr96\" (UniqueName: \"kubernetes.io/projected/e58e6ac7-8ee6-4790-8625-54d10183203e-kube-api-access-fmr96\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650861 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-node-log\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650370 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-lib-modules\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysconfig\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650522 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-var-lib-kubelet\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650284 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysctl-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.650584 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650734 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65c98535-b722-4529-b0b6-efdd4138b9ae-ovnkube-script-lib\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650748 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-kubelet\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.651002 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650437 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-modprobe-d\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.651040 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:23:50.151018126 +0000 UTC m=+3.132951975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:49.651062 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:50.151050951 +0000 UTC m=+3.132984788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651064 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-sysconfig\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.651740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650799 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-cni-netd\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650617 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-systemd-units\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58e6ac7-8ee6-4790-8625-54d10183203e-tmp-dir\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-tmp\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7thm\" (UniqueName: \"kubernetes.io/projected/ed3fbfbc-9a18-416a-bc05-f53bafee5058-kube-api-access-t7thm\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651241 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-run-openvswitch\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650932 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-sys\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650176 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65c98535-b722-4529-b0b6-efdd4138b9ae-host-run-netns\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.650147 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed3fbfbc-9a18-416a-bc05-f53bafee5058-run\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651631 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58e6ac7-8ee6-4790-8625-54d10183203e-tmp-dir\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.652646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.651971 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65c98535-b722-4529-b0b6-efdd4138b9ae-ovn-node-metrics-cert\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.653947 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:49.653903 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6af0dfb_4a51_4173_8611_6b091dd6751c.slice/crio-4597f6524b4ee7402dc3f2df92be41632a6a50de39c3208e0803b359999e8c98 WatchSource:0}: Error finding container 4597f6524b4ee7402dc3f2df92be41632a6a50de39c3208e0803b359999e8c98: Status 404 returned error can't find the container with id 4597f6524b4ee7402dc3f2df92be41632a6a50de39c3208e0803b359999e8c98 Jul 07 04:23:49.654431 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.654406 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-tmp\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.656400 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.656093 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed3fbfbc-9a18-416a-bc05-f53bafee5058-etc-tuned\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.661726 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:49.661700 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8888cdf0_ef1e_485f_83e7_80ac2797e99c.slice/crio-6f74529e88eaf238e1ebf4f6bc41f7b602cc40b3809a8a67cff72c174b4b6e21 WatchSource:0}: Error finding container 6f74529e88eaf238e1ebf4f6bc41f7b602cc40b3809a8a67cff72c174b4b6e21: Status 404 returned error can't find the container with id 6f74529e88eaf238e1ebf4f6bc41f7b602cc40b3809a8a67cff72c174b4b6e21 Jul 07 04:23:49.674629 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.674578 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqwv\" (UniqueName: \"kubernetes.io/projected/65c98535-b722-4529-b0b6-efdd4138b9ae-kube-api-access-tfqwv\") pod \"ovnkube-node-bhbbm\" (UID: \"65c98535-b722-4529-b0b6-efdd4138b9ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.676483 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.676286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:23:49.676483 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.676472 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7thm\" (UniqueName: \"kubernetes.io/projected/ed3fbfbc-9a18-416a-bc05-f53bafee5058-kube-api-access-t7thm\") pod \"tuned-h54j2\" (UID: \"ed3fbfbc-9a18-416a-bc05-f53bafee5058\") " pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:49.676702 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.676659 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmfg\" (UniqueName: \"kubernetes.io/projected/137a0240-63ec-41ac-910d-aafdbcfc59f0-kube-api-access-kbmfg\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:49.677883 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.677862 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr96\" (UniqueName: \"kubernetes.io/projected/e58e6ac7-8ee6-4790-8625-54d10183203e-kube-api-access-fmr96\") pod \"node-resolver-gvmr2\" (UID: \"e58e6ac7-8ee6-4790-8625-54d10183203e\") " pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.687514 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:49.687483 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c98535_b722_4529_b0b6_efdd4138b9ae.slice/crio-ef3cb3ab1eacfbceab3082900c9261b9366ff40c8591862308c34bfb91765b5e WatchSource:0}: Error finding container ef3cb3ab1eacfbceab3082900c9261b9366ff40c8591862308c34bfb91765b5e: Status 404 returned error can't find the container with id ef3cb3ab1eacfbceab3082900c9261b9366ff40c8591862308c34bfb91765b5e Jul 07 04:23:49.862963 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.862931 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vdnwl" Jul 07 04:23:49.959017 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.958933 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvmr2" Jul 07 04:23:49.970061 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:49.970034 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h54j2" Jul 07 04:23:50.156772 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.156732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:50.156943 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.156782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:50.156943 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.156854 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157267 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157387 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:23:51.157357263 +0000 UTC m=+4.139291100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157553 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157582 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157600 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157737 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:51.157719167 +0000 UTC m=+4.139653004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157790 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:50.159904 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.157921 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:51.157884386 +0000 UTC m=+4.139818219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:50.227599 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.227517 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:23:50.521917 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.521360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:50.521917 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:50.521503 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:50.547678 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.547563 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"ef3cb3ab1eacfbceab3082900c9261b9366ff40c8591862308c34bfb91765b5e"} Jul 07 04:23:50.551765 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.551731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" event={"ID":"8888cdf0-ef1e-485f-83e7-80ac2797e99c","Type":"ContainerStarted","Data":"6f74529e88eaf238e1ebf4f6bc41f7b602cc40b3809a8a67cff72c174b4b6e21"} Jul 07 04:23:50.558104 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.558033 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l26zw" event={"ID":"c6af0dfb-4a51-4173-8611-6b091dd6751c","Type":"ContainerStarted","Data":"4597f6524b4ee7402dc3f2df92be41632a6a50de39c3208e0803b359999e8c98"} Jul 07 04:23:50.562848 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.562813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerStarted","Data":"cf51fd5b2d565aa72f31acff12a172228a22ebf28f0c09f0c0c549e54c8aaa74"} Jul 07 04:23:50.564577 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.564518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c8tf5" event={"ID":"42292ea1-1500-4541-9321-06324275ce72","Type":"ContainerStarted","Data":"cb07aff70cbf425c7922e4e73b898c17437c1b6353d7c57c8cc801db2e720c5c"} Jul 07 04:23:50.567343 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.567317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvhjd" event={"ID":"71128bfb-74e3-45f7-a7c1-eca531c3cad8","Type":"ContainerStarted","Data":"8980a3db6bd9eb4266f44b8c681eab98eb067c24a3a5314b710b988188f3c2d5"} Jul 07 04:23:50.624364 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.624255 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 04:18:50 +0000 UTC" deadline="2027-12-31 17:01:51.273305702 +0000 UTC" Jul 07 04:23:50.624364 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:50.624311 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13020h38m0.648998695s" Jul 07 04:23:51.165014 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.164975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:51.165225 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.165042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:51.165225 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.165113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:51.165354 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165242 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:51.165354 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165308 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:23:53.165287468 +0000 UTC m=+6.147221319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:51.165469 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165383 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:51.165469 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165421 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:53.165407369 +0000 UTC m=+6.147341204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:51.165577 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165558 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:23:51.165629 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165599 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:23:51.165681 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165633 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:51.165733 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.165679 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:53.165665398 +0000 UTC m=+6.147599232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:51.524035 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.524002 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:51.524548 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.524147 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:51.524616 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.524575 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:51.524700 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:51.524675 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:51.625303 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.625258 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 04:18:50 +0000 UTC" deadline="2028-01-09 12:46:10.829835823 +0000 UTC" Jul 07 04:23:51.625303 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:51.625301 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13232h22m19.204539967s" Jul 07 04:23:51.749060 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:51.749030 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58e6ac7_8ee6_4790_8625_54d10183203e.slice/crio-8cde66aa9e7105260aeda3a7bb3e91baecbec33d81eb702d70719dfb0324b995 WatchSource:0}: Error finding container 8cde66aa9e7105260aeda3a7bb3e91baecbec33d81eb702d70719dfb0324b995: Status 404 returned error can't find the container with id 8cde66aa9e7105260aeda3a7bb3e91baecbec33d81eb702d70719dfb0324b995 Jul 07 04:23:51.750620 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:23:51.750579 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3fbfbc_9a18_416a_bc05_f53bafee5058.slice/crio-891f86e27fff6b86dac50988635115b3201e4ac5be3334734f5b50677b6c70e1 WatchSource:0}: Error finding container 891f86e27fff6b86dac50988635115b3201e4ac5be3334734f5b50677b6c70e1: Status 404 returned error can't find the container with id 891f86e27fff6b86dac50988635115b3201e4ac5be3334734f5b50677b6c70e1 Jul 07 04:23:52.521708 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:52.521670 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:52.521887 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:52.521848 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:52.575452 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:52.575416 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vdnwl" event={"ID":"d38057ab-6fb0-4a71-b834-79b76eaea8d8","Type":"ContainerStarted","Data":"bf4f003305be4fa1daf6406c4d479416909a1c7430191a994d8ed5eb119e4ca9"} Jul 07 04:23:52.579205 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:52.579174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvmr2" event={"ID":"e58e6ac7-8ee6-4790-8625-54d10183203e","Type":"ContainerStarted","Data":"8cde66aa9e7105260aeda3a7bb3e91baecbec33d81eb702d70719dfb0324b995"} Jul 07 04:23:52.580894 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:52.580865 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h54j2" event={"ID":"ed3fbfbc-9a18-416a-bc05-f53bafee5058","Type":"ContainerStarted","Data":"891f86e27fff6b86dac50988635115b3201e4ac5be3334734f5b50677b6c70e1"} Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:53.183608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:53.183677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:53.183751 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.183897 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.183964 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:23:57.183941695 +0000 UTC m=+10.165875546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184372 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184432 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:57.184414612 +0000 UTC m=+10.166348466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184514 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184529 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184541 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:53.184649 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.184586 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:23:57.184575548 +0000 UTC m=+10.166509379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:53.523639 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:53.523611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:53.523815 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:53.523618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:53.523815 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.523784 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:53.523946 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:53.523915 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:54.521930 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:54.521887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:54.522404 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:54.522018 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:55.522027 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:55.521915 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:55.522529 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:55.522048 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:55.522529 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:55.522207 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:55.522529 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:55.522340 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:56.522026 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:56.521992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:56.522206 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:56.522147 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:57.219906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220056 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:57.220082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220114 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220129 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220186 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.220166756 +0000 UTC m=+18.202100611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220202 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220227 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:57.220126 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:57.220269 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220238 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.220226095 +0000 UTC m=+18.202159938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:23:57.220929 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.220312 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.220293417 +0000 UTC m=+18.202227253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:23:57.523469 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:57.523435 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:57.523943 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.523570 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:57.524760 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:57.524736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:57.524923 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:57.524895 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:58.522481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:58.522344 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:23:58.522631 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:58.522480 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:23:58.608386 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:58.608347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" event={"ID":"44f23d15227f2b810bc3ab580d4a5aaa","Type":"ContainerStarted","Data":"99d42e323ac41e194b4424b7d2d09f6bedc23c3bffe71f34eacf84e6cf83458b"} Jul 07 04:23:58.625263 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:58.625210 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" podStartSLOduration=9.625175534 podStartE2EDuration="9.625175534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:23:58.624963245 +0000 UTC m=+11.606897101" watchObservedRunningTime="2026-07-07 04:23:58.625175534 +0000 UTC m=+11.607109386" Jul 07 04:23:59.522524 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.522389 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:23:59.522677 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:59.522533 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:23:59.522677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.522589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:23:59.522828 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:23:59.522705 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:23:59.612782 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.612731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c8tf5" event={"ID":"42292ea1-1500-4541-9321-06324275ce72","Type":"ContainerStarted","Data":"88d731c0350489a24ef988543984253d3a4df1666843a286e7c32e7fb34e1b70"} Jul 07 04:23:59.615009 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.614978 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvhjd" event={"ID":"71128bfb-74e3-45f7-a7c1-eca531c3cad8","Type":"ContainerStarted","Data":"0d7e424ded0db8488b7c81cee34c35ee612fb1f3d89b92b781fc9be2db5c5eb1"} Jul 07 04:23:59.617487 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.617464 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvmr2" event={"ID":"e58e6ac7-8ee6-4790-8625-54d10183203e","Type":"ContainerStarted","Data":"7ada65c4660ee8e48cb1b6f33b61c156cfe9802b9ed71ab6f3a3dfb325e54286"} Jul 07 04:23:59.619685 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.619660 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f00651057cf051b0e8c326f620be66f" containerID="179dcca433bb7edc4ecc4fc57ca6f6d115f7f2b6567015ed744b9f8b867527e2" exitCode=0 Jul 07 04:23:59.619783 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.619734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"5f00651057cf051b0e8c326f620be66f","Type":"ContainerDied","Data":"179dcca433bb7edc4ecc4fc57ca6f6d115f7f2b6567015ed744b9f8b867527e2"} Jul 07 04:23:59.622543 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.622517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" event={"ID":"8888cdf0-ef1e-485f-83e7-80ac2797e99c","Type":"ContainerStarted","Data":"4dc4d2db3874717d7e760edf2132fe705ed9a178479be5d05aecd5cef9e83ee5"} Jul 07 04:23:59.624438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.624410 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerStarted","Data":"3c6fa6c84b20b27193a94d9a84cd4865cfaed5fce24f342d4b7f3acb30b36947"} Jul 07 04:23:59.628655 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.628610 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c8tf5" podStartSLOduration=3.004415762 podStartE2EDuration="11.628595434s" totalImagesPullingTime="8.624179672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:23:59.628361577 +0000 UTC m=+12.610295434" watchObservedRunningTime="2026-07-07 04:23:59.628595434 +0000 UTC m=+12.610529291" Jul 07 04:23:59.673380 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.672906 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mvhjd" podStartSLOduration=3.039968096 podStartE2EDuration="11.672890657s" totalImagesPullingTime="8.632922561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:23:59.672702306 +0000 UTC m=+12.654636160" watchObservedRunningTime="2026-07-07 04:23:59.672890657 +0000 UTC m=+12.654824508" Jul 07 04:23:59.703436 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:23:59.703350 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gvmr2" podStartSLOduration=5.124312012 podStartE2EDuration="11.703334298s" totalImagesPullingTime="6.579022286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:23:59.687077254 +0000 UTC m=+12.669011108" watchObservedRunningTime="2026-07-07 04:23:59.703334298 +0000 UTC m=+12.685268144" Jul 07 04:24:00.521861 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:00.521829 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:00.522042 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:00.521962 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:00.627810 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:00.627758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vdnwl" event={"ID":"d38057ab-6fb0-4a71-b834-79b76eaea8d8","Type":"ContainerStarted","Data":"6e9f45096149bf1cbc0bf59b299a9315d929be4aeb1047601832e6591bbfa2ad"} Jul 07 04:24:01.521876 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:01.521841 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:01.522044 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:01.521880 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:01.522044 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:01.521976 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:01.522164 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:01.522122 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:01.631246 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:01.631163 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="3c6fa6c84b20b27193a94d9a84cd4865cfaed5fce24f342d4b7f3acb30b36947" exitCode=0 Jul 07 04:24:01.631246 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:01.631208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"3c6fa6c84b20b27193a94d9a84cd4865cfaed5fce24f342d4b7f3acb30b36947"} Jul 07 04:24:01.654916 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:01.654874 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vdnwl" podStartSLOduration=7.056198132 podStartE2EDuration="13.654860529s" totalImagesPullingTime="6.598662397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:00.641916756 +0000 UTC m=+13.623850609" watchObservedRunningTime="2026-07-07 04:24:01.654860529 +0000 UTC m=+14.636794386" Jul 07 04:24:02.521541 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:02.521512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:02.521733 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:02.521631 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:03.338272 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.338234 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:24:03.338864 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.338846 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:24:03.521484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.521439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:03.521648 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:03.521562 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:03.521648 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.521622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:03.521800 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:03.521739 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:03.634702 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.634627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:24:03.635177 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:03.635152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mvhjd" Jul 07 04:24:04.521342 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:04.521318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:04.521880 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:04.521425 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:05.282036 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:05.282002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:05.282202 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:05.282058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:05.282202 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:05.282125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:05.282202 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282144 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.282202 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282190 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:05.282202 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282199 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.282457 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282217 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:05.282457 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282205 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:21.282186597 +0000 UTC m=+34.264120450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.282457 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282230 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:05.282457 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282246 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:24:21.282235574 +0000 UTC m=+34.264169411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.282457 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.282287 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:21.282273166 +0000 UTC m=+34.264207000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:05.524318 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:05.524291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:05.524638 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.524378 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:05.524638 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:05.524454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:05.524638 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:05.524537 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:06.522086 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:06.522006 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:06.522264 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:06.522138 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:07.523323 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:07.523278 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:07.523828 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:07.523379 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:07.523828 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:07.523409 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:07.523828 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:07.523511 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:08.521813 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:08.521763 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:08.521974 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:08.521900 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:09.523145 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.523070 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:09.525686 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:09.524508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:09.533596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.528065 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:09.533596 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:09.528267 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:09.608332 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.608309 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 04:24:09.648737 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.648708 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"d2470ebd7020d77d1f2aad5a6e804c41d4585a55789bc668bd67d83c0e58b961"} Jul 07 04:24:09.648873 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.648851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"012958fd169e3fc8174ae1f43a2107c86a0158b60d591e9ed9c96a621c78d844"} Jul 07 04:24:09.650883 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.650785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h54j2" event={"ID":"ed3fbfbc-9a18-416a-bc05-f53bafee5058","Type":"ContainerStarted","Data":"2ae884da7582053a16a6176980ace51b376669b2c4062a4d07f4cd0c9dbdf263"} Jul 07 04:24:09.654279 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.654257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"5f00651057cf051b0e8c326f620be66f","Type":"ContainerStarted","Data":"fc15b9845fed2592fb1c40f77810649a7f43d4d9d0fe6ed1064f0b526018ec26"} Jul 07 04:24:09.657373 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.657347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" event={"ID":"8888cdf0-ef1e-485f-83e7-80ac2797e99c","Type":"ContainerStarted","Data":"b22811715a32bb9cfce31d2cbca95915d8c25284a28f035e4eb174fc8145da67"} Jul 07 04:24:09.659452 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.659429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l26zw" event={"ID":"c6af0dfb-4a51-4173-8611-6b091dd6751c","Type":"ContainerStarted","Data":"1c0481d4532bdfa04aa6796c0a4120963f18fae73052075d15bf42df73b664e4"} Jul 07 04:24:09.683933 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.683887 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h54j2" podStartSLOduration=4.198529124 podStartE2EDuration="21.683872034s" totalImagesPullingTime="17.48534291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:09.670319065 +0000 UTC m=+22.652252922" watchObservedRunningTime="2026-07-07 04:24:09.683872034 +0000 UTC m=+22.665805865" Jul 07 04:24:09.684346 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.684309 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" podStartSLOduration=20.684297684 podStartE2EDuration="20.684297684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:09.683993072 +0000 UTC m=+22.665926925" watchObservedRunningTime="2026-07-07 04:24:09.684297684 +0000 UTC m=+22.666231541" Jul 07 04:24:09.717757 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:09.717709 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l26zw" podStartSLOduration=1.973958089 podStartE2EDuration="21.717693915s" totalImagesPullingTime="19.743735826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:09.717678901 +0000 UTC m=+22.699612752" watchObservedRunningTime="2026-07-07 04:24:09.717693915 +0000 UTC m=+22.699627766" Jul 07 04:24:10.520829 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.520591 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T04:24:09.608330009Z","UUID":"08ffa5fd-3ad2-4ede-ad30-2423b72ac8db","Handler":null,"Name":"","Endpoint":""} Jul 07 04:24:10.521459 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.521432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:10.521569 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:10.521544 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:10.522461 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.522440 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 04:24:10.522584 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.522484 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 04:24:10.664402 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.664365 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"c99ba4f06a88b6355c1c4e03ace5105fc8c14b6bf11d3c355d40efbe4371309d"} Jul 07 04:24:10.664819 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.664448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"eca7ebebc692b1ed82cee0c83adc6e7e8285536881b187734e191413b788f5c1"} Jul 07 04:24:10.664819 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.664481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"ae965ec6f850a52efa54004b57f1b5149ed2d6703b05b68640b838f2a9841904"} Jul 07 04:24:10.664819 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:10.664510 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"b39d7e4e2f2ffac82a7f7104e7d022af169ae0f48a69f1e067b97da22ce3a673"} Jul 07 04:24:11.522439 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:11.522403 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:11.522743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:11.522403 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:11.522743 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:11.522515 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:11.522743 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:11.522572 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:11.668153 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:11.668117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" event={"ID":"8888cdf0-ef1e-485f-83e7-80ac2797e99c","Type":"ContainerStarted","Data":"e78434c2e1d47e8025824f55c3c1026bdb0ab1e8cc2fe9178e3a5551faa762b8"} Jul 07 04:24:11.685670 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:11.685631 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v749r" podStartSLOduration=2.983416493 podStartE2EDuration="23.685619431s" totalImagesPullingTime="20.702202938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:11.685497069 +0000 UTC m=+24.667430918" watchObservedRunningTime="2026-07-07 04:24:11.685619431 +0000 UTC m=+24.667553282" Jul 07 04:24:12.521665 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:12.521626 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:12.521855 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:12.521732 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:12.672810 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:12.672779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"814d52d56a2242ee23c8c9fffe315856ec221500ead4928966ca1d1d6c4ffb77"} Jul 07 04:24:13.521786 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:13.521745 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:13.521964 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:13.521867 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:13.522191 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:13.521751 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:13.522191 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:13.522159 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:14.521704 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:14.521674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:14.522398 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:14.521807 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:14.682588 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:14.682536 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" event={"ID":"65c98535-b722-4529-b0b6-efdd4138b9ae","Type":"ContainerStarted","Data":"25b2492df2186efa8d3ede06aa4e10c6539529ea6c31088db5403dd7dd8bb9bf"} Jul 07 04:24:14.682999 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:14.682867 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:14.699653 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:14.699630 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:14.710355 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:14.710308 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" podStartSLOduration=6.998828892 podStartE2EDuration="26.710291947s" totalImagesPullingTime="19.711463055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:14.709697507 +0000 UTC m=+27.691631359" watchObservedRunningTime="2026-07-07 04:24:14.710291947 +0000 UTC m=+27.692225799" Jul 07 04:24:15.525409 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:15.525380 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:15.525798 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:15.525506 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:15.525798 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:15.525555 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:15.525798 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:15.525658 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:15.686281 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:15.686139 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:15.686281 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:15.686223 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:15.701899 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:15.701873 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:16.144497 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.144473 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bn4nk"] Jul 07 04:24:16.144611 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.144580 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:16.144704 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:16.144685 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:16.148302 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.148277 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tvqx"] Jul 07 04:24:16.148411 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.148383 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:16.148505 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:16.148483 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:16.149016 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.148991 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7mhwl"] Jul 07 04:24:16.149107 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.149094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:16.149222 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:16.149196 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:16.689009 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.688981 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="743640c2f2df76533b18d3dca54beb719a9152b8866e10dd12323892810cc01a" exitCode=0 Jul 07 04:24:16.689349 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:16.689058 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"743640c2f2df76533b18d3dca54beb719a9152b8866e10dd12323892810cc01a"} Jul 07 04:24:17.522929 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:17.522896 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:17.523083 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:17.522996 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:18.522115 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:18.522089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:18.522431 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:18.522089 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:18.522431 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:18.522181 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:18.522431 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:18.522262 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:18.695649 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:18.695617 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="f79a62271fc3f0fe280cc7489204a17a7d036400fc183e1e9a4adaa7c1180e02" exitCode=0 Jul 07 04:24:18.695811 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:18.695689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"f79a62271fc3f0fe280cc7489204a17a7d036400fc183e1e9a4adaa7c1180e02"} Jul 07 04:24:19.521976 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:19.521793 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:19.522155 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:19.522057 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:19.700097 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:19.700069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerStarted","Data":"cb92613cea696dc55d69323d4c33b194a7f0eceb15706662f97730c1588fbdcb"} Jul 07 04:24:20.521693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:20.521663 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:20.521859 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:20.521785 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tvqx" podUID="8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca" Jul 07 04:24:20.521859 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:20.521798 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:20.521947 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:20.521907 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7mhwl" podUID="137a0240-63ec-41ac-910d-aafdbcfc59f0" Jul 07 04:24:20.704605 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:20.704579 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="cb92613cea696dc55d69323d4c33b194a7f0eceb15706662f97730c1588fbdcb" exitCode=0 Jul 07 04:24:20.705052 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:20.704627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"cb92613cea696dc55d69323d4c33b194a7f0eceb15706662f97730c1588fbdcb"} Jul 07 04:24:21.303781 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:21.303734 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:21.303802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:21.303832 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303893 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303907 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303913 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303925 2563 projected.go:196] Error preparing data for projected volume kube-api-access-g5w5r for pod openshift-network-diagnostics/network-check-target-bn4nk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:21.303935 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303928 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:21.304179 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303962 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs podName:137a0240-63ec-41ac-910d-aafdbcfc59f0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:53.303945021 +0000 UTC m=+66.285878850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs") pod "network-metrics-daemon-7mhwl" (UID: "137a0240-63ec-41ac-910d-aafdbcfc59f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:21.304179 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303976 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r podName:71b154b2-7eca-4ccb-a50c-ab875f43daa0 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:53.303968764 +0000 UTC m=+66.285902594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5w5r" (UniqueName: "kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r") pod "network-check-target-bn4nk" (UID: "71b154b2-7eca-4ccb-a50c-ab875f43daa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:21.304179 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.303986 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret podName:8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca nodeName:}" failed. No retries permitted until 2026-07-07 04:24:53.303981509 +0000 UTC m=+66.285915339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret") pod "global-pull-secret-syncer-2tvqx" (UID: "8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:21.522327 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:21.522296 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:21.522498 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:21.522419 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bn4nk" podUID="71b154b2-7eca-4ccb-a50c-ab875f43daa0" Jul 07 04:24:22.346424 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.346394 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeReady" Jul 07 04:24:22.346915 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.346527 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 04:24:22.382179 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.382152 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv"] Jul 07 04:24:22.403891 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.403838 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cp2fz"] Jul 07 04:24:22.404132 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.404113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" Jul 07 04:24:22.407327 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.407287 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.408655 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.408277 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rtqj4\"" Jul 07 04:24:22.408655 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.408538 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.421280 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.421262 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:24:22.421403 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.421387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.424361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.424256 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 04:24:22.424361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.424315 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 04:24:22.424361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.424317 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.424705 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.424685 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bzg9w\"" Jul 07 04:24:22.425347 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.425273 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.430355 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.430336 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 04:24:22.439411 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.439383 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4"] Jul 07 04:24:22.439524 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.439512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.442071 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.442022 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 04:24:22.442071 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.442061 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 04:24:22.442197 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.442110 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6jvd\"" Jul 07 04:24:22.442336 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.442321 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 04:24:22.457481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.457462 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz"] Jul 07 04:24:22.457605 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.457589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.460317 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.460297 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 04:24:22.460410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.460366 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.460410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.460394 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4k8g7\"" Jul 07 04:24:22.460491 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.460417 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.462171 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.462120 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 04:24:22.469639 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.469621 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs"] Jul 07 04:24:22.469743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.469723 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.472108 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.472088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.472108 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.472088 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 04:24:22.472257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.472119 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-24c7j\"" Jul 07 04:24:22.472257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.472152 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 04:24:22.472257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.472088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.481637 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481620 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv"] Jul 07 04:24:22.481741 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481652 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4"] Jul 07 04:24:22.481741 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481674 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz"] Jul 07 04:24:22.481741 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481693 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cp2fz"] Jul 07 04:24:22.481741 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481715 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs"] Jul 07 04:24:22.481741 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481727 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.481953 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.481733 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv"] Jul 07 04:24:22.484996 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.484978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 04:24:22.485087 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.484985 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 04:24:22.485246 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.485221 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.485343 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.485314 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.485443 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.485422 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q66fg\"" Jul 07 04:24:22.493787 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.493751 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:24:22.493862 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.493800 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pkxhb"] Jul 07 04:24:22.493862 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.493845 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" Jul 07 04:24:22.495853 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.495836 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.495938 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.495914 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kj9kp\"" Jul 07 04:24:22.496024 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.496009 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.507929 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.507912 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-72lzq"] Jul 07 04:24:22.508020 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.508007 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.509938 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.509921 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jlfpx\"" Jul 07 04:24:22.510020 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.509959 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 04:24:22.510020 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.509994 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 04:24:22.510163 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.510022 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 04:24:22.511085 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.511066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g74k\" (UniqueName: \"kubernetes.io/projected/cab9ef11-19ee-4259-9fb0-f68c45be6f75-kube-api-access-7g74k\") pod \"volume-data-source-validator-5d456fdbcd-s9hlv\" (UID: \"cab9ef11-19ee-4259-9fb0-f68c45be6f75\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" Jul 07 04:24:22.521968 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.521951 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv"] Jul 07 04:24:22.522066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.521987 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72lzq"] Jul 07 04:24:22.522066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.522014 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkxhb"] Jul 07 04:24:22.522066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.522031 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.522185 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.522100 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:22.522185 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.522031 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:22.524875 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.524858 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qn6gg\"" Jul 07 04:24:22.525059 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.525046 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 04:24:22.525168 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.525152 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 04:24:22.525229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.525177 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 04:24:22.525229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.525182 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krb48\"" Jul 07 04:24:22.525229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.525192 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 04:24:22.611621 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.611621 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ck7r\" (UniqueName: \"kubernetes.io/projected/a76e076d-8cd5-4e3c-9223-653a53e02f89-kube-api-access-6ck7r\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.611801 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g74k\" (UniqueName: \"kubernetes.io/projected/cab9ef11-19ee-4259-9fb0-f68c45be6f75-kube-api-access-7g74k\") pod \"volume-data-source-validator-5d456fdbcd-s9hlv\" (UID: \"cab9ef11-19ee-4259-9fb0-f68c45be6f75\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" Jul 07 04:24:22.611801 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611670 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c61aea7-54b3-4024-8f7d-5b1a0d806617-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.611912 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-trusted-ca\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.611912 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611868 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-config\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.611912 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55vq\" (UniqueName: \"kubernetes.io/projected/a92b02e2-395b-44df-89ec-8d335b6e26fc-kube-api-access-n55vq\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.612076 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612076 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.611986 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8aac71c4-980e-4b85-a65f-e40c93af620c-cert\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.612076 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612013 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a76e076d-8cd5-4e3c-9223-653a53e02f89-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.612076 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6rj\" (UniqueName: \"kubernetes.io/projected/42649923-329b-4b07-a064-f5c1ba227943-kube-api-access-8b6rj\") pod \"network-check-source-989f8b8dc-jnmnv\" (UID: \"42649923-329b-4b07-a064-f5c1ba227943\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a92b02e2-395b-44df-89ec-8d335b6e26fc-serving-cert\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612141 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c61aea7-54b3-4024-8f7d-5b1a0d806617-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85gm\" (UniqueName: \"kubernetes.io/projected/2c61aea7-54b3-4024-8f7d-5b1a0d806617-kube-api-access-x85gm\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.612257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612269 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ts7j\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqhn\" (UniqueName: \"kubernetes.io/projected/729c14bc-b79b-4ddc-8a04-11a2fef542ab-kube-api-access-jzqhn\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a76e076d-8cd5-4e3c-9223-653a53e02f89-config\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.612568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.612400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5q9\" (UniqueName: \"kubernetes.io/projected/8aac71c4-980e-4b85-a65f-e40c93af620c-kube-api-access-6j5q9\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.621952 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.621911 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g74k\" (UniqueName: \"kubernetes.io/projected/cab9ef11-19ee-4259-9fb0-f68c45be6f75-kube-api-access-7g74k\") pod \"volume-data-source-validator-5d456fdbcd-s9hlv\" (UID: \"cab9ef11-19ee-4259-9fb0-f68c45be6f75\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" Jul 07 04:24:22.713007 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.712984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c61aea7-54b3-4024-8f7d-5b1a0d806617-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.713088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-config\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.713088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8aac71c4-980e-4b85-a65f-e40c93af620c-cert\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.713088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6rj\" (UniqueName: \"kubernetes.io/projected/42649923-329b-4b07-a064-f5c1ba227943-kube-api-access-8b6rj\") pod \"network-check-source-989f8b8dc-jnmnv\" (UID: \"42649923-329b-4b07-a064-f5c1ba227943\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" Jul 07 04:24:22.713088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713109 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a92b02e2-395b-44df-89ec-8d335b6e26fc-serving-cert\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fc374-e44f-4652-9308-c1b1036a56cc-config-volume\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ts7j\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713266 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqhn\" (UniqueName: \"kubernetes.io/projected/729c14bc-b79b-4ddc-8a04-11a2fef542ab-kube-api-access-jzqhn\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.713321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a76e076d-8cd5-4e3c-9223-653a53e02f89-config\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5q9\" (UniqueName: \"kubernetes.io/projected/8aac71c4-980e-4b85-a65f-e40c93af620c-kube-api-access-6j5q9\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-trusted-ca\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55vq\" (UniqueName: \"kubernetes.io/projected/a92b02e2-395b-44df-89ec-8d335b6e26fc-kube-api-access-n55vq\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a76e076d-8cd5-4e3c-9223-653a53e02f89-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713575 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c61aea7-54b3-4024-8f7d-5b1a0d806617-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ck7r\" (UniqueName: \"kubernetes.io/projected/a76e076d-8cd5-4e3c-9223-653a53e02f89-kube-api-access-6ck7r\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713662 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85gm\" (UniqueName: \"kubernetes.io/projected/2c61aea7-54b3-4024-8f7d-5b1a0d806617-kube-api-access-x85gm\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.713693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3fc374-e44f-4652-9308-c1b1036a56cc-tmp-dir\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.714243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713723 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvbt\" (UniqueName: \"kubernetes.io/projected/1f3fc374-e44f-4652-9308-c1b1036a56cc-kube-api-access-zgvbt\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.714243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.713842 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-config\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.715320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.714655 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.715320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.714681 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a76e076d-8cd5-4e3c-9223-653a53e02f89-config\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.715320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715006 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.715320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715244 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c61aea7-54b3-4024-8f7d-5b1a0d806617-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715552 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a92b02e2-395b-44df-89ec-8d335b6e26fc-trusted-ca\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715807 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c61aea7-54b3-4024-8f7d-5b1a0d806617-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.715837 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.715852 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.715917 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:23.215897647 +0000 UTC m=+36.197831478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715944 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715944 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a92b02e2-395b-44df-89ec-8d335b6e26fc-serving-cert\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.715962 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.716042 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:22.716319 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.716090 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:23.216075969 +0000 UTC m=+36.198009803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:22.716746 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.716497 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" Jul 07 04:24:22.716746 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.716498 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8aac71c4-980e-4b85-a65f-e40c93af620c-cert\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.716746 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.716639 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a76e076d-8cd5-4e3c-9223-653a53e02f89-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.717873 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.717854 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.722584 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.722564 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55vq\" (UniqueName: \"kubernetes.io/projected/a92b02e2-395b-44df-89ec-8d335b6e26fc-kube-api-access-n55vq\") pod \"console-operator-68b757fb67-cp2fz\" (UID: \"a92b02e2-395b-44df-89ec-8d335b6e26fc\") " pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.722667 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.722651 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6rj\" (UniqueName: \"kubernetes.io/projected/42649923-329b-4b07-a064-f5c1ba227943-kube-api-access-8b6rj\") pod \"network-check-source-989f8b8dc-jnmnv\" (UID: \"42649923-329b-4b07-a064-f5c1ba227943\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" Jul 07 04:24:22.729824 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.727055 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ts7j\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.729824 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.727545 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85gm\" (UniqueName: \"kubernetes.io/projected/2c61aea7-54b3-4024-8f7d-5b1a0d806617-kube-api-access-x85gm\") pod \"kube-storage-version-migrator-operator-846bf668f7-wnvlz\" (UID: \"2c61aea7-54b3-4024-8f7d-5b1a0d806617\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.729824 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.727709 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqhn\" (UniqueName: \"kubernetes.io/projected/729c14bc-b79b-4ddc-8a04-11a2fef542ab-kube-api-access-jzqhn\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:22.729824 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.728918 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5q9\" (UniqueName: \"kubernetes.io/projected/8aac71c4-980e-4b85-a65f-e40c93af620c-kube-api-access-6j5q9\") pod \"ingress-canary-pkxhb\" (UID: \"8aac71c4-980e-4b85-a65f-e40c93af620c\") " pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.729824 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.728940 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ck7r\" (UniqueName: \"kubernetes.io/projected/a76e076d-8cd5-4e3c-9223-653a53e02f89-kube-api-access-6ck7r\") pod \"service-ca-operator-5d46d9ffd4-svjfs\" (UID: \"a76e076d-8cd5-4e3c-9223-653a53e02f89\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.731683 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.731655 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:22.732049 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.732027 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:22.779875 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.779848 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" Jul 07 04:24:22.790025 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.790003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" Jul 07 04:24:22.801255 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.801235 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.814865 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fc374-e44f-4652-9308-c1b1036a56cc-config-volume\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.814994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3fc374-e44f-4652-9308-c1b1036a56cc-tmp-dir\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.815033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvbt\" (UniqueName: \"kubernetes.io/projected/1f3fc374-e44f-4652-9308-c1b1036a56cc-kube-api-access-zgvbt\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.815078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.815109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkxhb" Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.815199 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:22.815302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:23.315283503 +0000 UTC m=+36.297217338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:22.815416 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.815335 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3fc374-e44f-4652-9308-c1b1036a56cc-tmp-dir\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.815838 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.815514 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fc374-e44f-4652-9308-c1b1036a56cc-config-volume\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.824402 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.824356 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvbt\" (UniqueName: \"kubernetes.io/projected/1f3fc374-e44f-4652-9308-c1b1036a56cc-kube-api-access-zgvbt\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:22.943403 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.943054 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cp2fz"] Jul 07 04:24:22.968405 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:22.968326 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv"] Jul 07 04:24:23.006555 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.002563 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz"] Jul 07 04:24:23.033435 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:23.033401 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c61aea7_54b3_4024_8f7d_5b1a0d806617.slice/crio-4e2358a339c7df489bb7edf3543edfd24fb08998922d400e8f6591bafc8bda99 WatchSource:0}: Error finding container 4e2358a339c7df489bb7edf3543edfd24fb08998922d400e8f6591bafc8bda99: Status 404 returned error can't find the container with id 4e2358a339c7df489bb7edf3543edfd24fb08998922d400e8f6591bafc8bda99 Jul 07 04:24:23.045847 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.045564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs"] Jul 07 04:24:23.048705 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:23.048679 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76e076d_8cd5_4e3c_9223_653a53e02f89.slice/crio-b0be39fbc0085bfd392a5a7968c4574017b151841e3f85544c96282210e3ec73 WatchSource:0}: Error finding container b0be39fbc0085bfd392a5a7968c4574017b151841e3f85544c96282210e3ec73: Status 404 returned error can't find the container with id b0be39fbc0085bfd392a5a7968c4574017b151841e3f85544c96282210e3ec73 Jul 07 04:24:23.071615 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.071543 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkxhb"] Jul 07 04:24:23.074516 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:23.074478 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aac71c4_980e_4b85_a65f_e40c93af620c.slice/crio-9269eb33659faaccefab36f618536f6e8ba912b792b5405d58f88950d37ad2f4 WatchSource:0}: Error finding container 9269eb33659faaccefab36f618536f6e8ba912b792b5405d58f88950d37ad2f4: Status 404 returned error can't find the container with id 9269eb33659faaccefab36f618536f6e8ba912b792b5405d58f88950d37ad2f4 Jul 07 04:24:23.099869 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.099843 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv"] Jul 07 04:24:23.102859 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:23.102825 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42649923_329b_4b07_a064_f5c1ba227943.slice/crio-08faeb0b0e99f20dcd1f871e4bf03831ff701317eda8fec1a97c58bf110fdef5 WatchSource:0}: Error finding container 08faeb0b0e99f20dcd1f871e4bf03831ff701317eda8fec1a97c58bf110fdef5: Status 404 returned error can't find the container with id 08faeb0b0e99f20dcd1f871e4bf03831ff701317eda8fec1a97c58bf110fdef5 Jul 07 04:24:23.219568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.219511 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:23.219568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.219547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:23.219699 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.219649 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:23.219699 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.219686 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:23.219699 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.219696 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:23.219833 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.219704 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:24.219689874 +0000 UTC m=+37.201623707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:23.219833 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.219731 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:24.21971987 +0000 UTC m=+37.201653700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:23.320402 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.320377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:23.320543 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.320525 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:23.320593 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:23.320586 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:24.320566696 +0000 UTC m=+37.302500525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:23.525243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.525222 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:23.528011 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.527990 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-krw7t\"" Jul 07 04:24:23.713680 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.713644 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkxhb" event={"ID":"8aac71c4-980e-4b85-a65f-e40c93af620c","Type":"ContainerStarted","Data":"9269eb33659faaccefab36f618536f6e8ba912b792b5405d58f88950d37ad2f4"} Jul 07 04:24:23.715110 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.715081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" event={"ID":"a76e076d-8cd5-4e3c-9223-653a53e02f89","Type":"ContainerStarted","Data":"b0be39fbc0085bfd392a5a7968c4574017b151841e3f85544c96282210e3ec73"} Jul 07 04:24:23.716709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.716683 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" event={"ID":"42649923-329b-4b07-a064-f5c1ba227943","Type":"ContainerStarted","Data":"08faeb0b0e99f20dcd1f871e4bf03831ff701317eda8fec1a97c58bf110fdef5"} Jul 07 04:24:23.718561 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.718535 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" event={"ID":"cab9ef11-19ee-4259-9fb0-f68c45be6f75","Type":"ContainerStarted","Data":"32f49bada83420acc7c21f91c2d4c99293c10fa268ea7977c41f9880cb73438c"} Jul 07 04:24:23.719933 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.719909 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" event={"ID":"2c61aea7-54b3-4024-8f7d-5b1a0d806617","Type":"ContainerStarted","Data":"4e2358a339c7df489bb7edf3543edfd24fb08998922d400e8f6591bafc8bda99"} Jul 07 04:24:23.721576 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.721531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" event={"ID":"a92b02e2-395b-44df-89ec-8d335b6e26fc","Type":"ContainerStarted","Data":"390a1fc80d31e9b20da048c0e79bfa840b313ec7d8b9007980eb6e1fe3429149"} Jul 07 04:24:23.858944 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.858844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mvhjd_71128bfb-74e3-45f7-a7c1-eca531c3cad8/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:23.948404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:23.948351 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mvhjd_71128bfb-74e3-45f7-a7c1-eca531c3cad8/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:24.231558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:24.231613 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.231786 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.231801 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.231858 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:26.231839622 +0000 UTC m=+39.213773475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.232254 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:24.232405 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.232303 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:26.232287857 +0000 UTC m=+39.214221692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:24.332958 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:24.332150 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:24.332958 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.332398 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:24.332958 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:24.332458 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:26.332440935 +0000 UTC m=+39.314374768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:25.731665 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.731528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" event={"ID":"cab9ef11-19ee-4259-9fb0-f68c45be6f75","Type":"ContainerStarted","Data":"958ef177573783977e0b5c0ff20d6988088b4199d9ea020917c95720d169271f"} Jul 07 04:24:25.823574 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.823511 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-s9hlv" podStartSLOduration=30.80793999 podStartE2EDuration="32.823485425s" totalImagesPullingTime="2.015545435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.748820764 +0000 UTC m=+38.730754616" watchObservedRunningTime="2026-07-07 04:24:25.823485425 +0000 UTC m=+38.805419279" Jul 07 04:24:25.826239 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.826207 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-npwxk"] Jul 07 04:24:25.831486 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.831462 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:25.833469 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.833415 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-npwxk"] Jul 07 04:24:25.834193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.834103 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k5rdd\"" Jul 07 04:24:25.834483 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.834450 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 04:24:25.834980 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.834812 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 04:24:25.847481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.847285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/092433f8-30e4-4c3c-919c-901d32b4a390-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:25.847572 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.847530 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:25.948753 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.948719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/092433f8-30e4-4c3c-919c-901d32b4a390-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:25.948921 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.948814 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:25.948987 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:25.948968 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:25.949052 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:25.949043 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:26.449027606 +0000 UTC m=+39.430961435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:25.949438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:25.949412 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/092433f8-30e4-4c3c-919c-901d32b4a390-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:26.251211 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:26.251171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:26.251358 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:26.251230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:26.251413 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.251364 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:26.251461 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.251412 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:26.251461 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.251432 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:26.251559 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.251463 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:30.251439369 +0000 UTC m=+43.233373219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:26.251559 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.251486 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:30.251476238 +0000 UTC m=+43.233410068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:26.352261 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:26.352229 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:26.352438 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.352379 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:26.352496 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.352446 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:30.352428728 +0000 UTC m=+43.334362559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:26.453645 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:26.453606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:26.453845 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.453784 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:26.453908 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:26.453863 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:27.453842599 +0000 UTC m=+40.435776429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:27.462554 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:27.462514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:27.463031 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:27.462620 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:27.463031 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:27.462667 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:29.462654453 +0000 UTC m=+42.444588283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:29.477984 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:29.477953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:29.478407 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:29.478126 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:29.478407 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:29.478240 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:33.478218792 +0000 UTC m=+46.460152622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:30.284924 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:30.284888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:30.284924 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:30.284926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:30.285171 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.285063 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:30.285171 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.285070 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:30.285171 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.285154 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:38.285131741 +0000 UTC m=+51.267065592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:30.285171 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.285080 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:30.285388 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.285215 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:38.285199756 +0000 UTC m=+51.267133591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:30.385390 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:30.385360 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:30.385564 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.385470 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:30.385564 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:30.385532 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:38.385515246 +0000 UTC m=+51.367449078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:32.750087 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.750048 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" event={"ID":"42649923-329b-4b07-a064-f5c1ba227943","Type":"ContainerStarted","Data":"6bcbdb1f823ed18f8afc1e0f5dcd153ee10125f3bf49f8efe025aa74d680c5be"} Jul 07 04:24:32.753180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.753155 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="79399a8fd0df51dc8df2bc1e08bcc79947c1b06e65b4e2d447c6710d55e1d6a2" exitCode=0 Jul 07 04:24:32.753312 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.753227 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"79399a8fd0df51dc8df2bc1e08bcc79947c1b06e65b4e2d447c6710d55e1d6a2"} Jul 07 04:24:32.755421 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.755398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" event={"ID":"2c61aea7-54b3-4024-8f7d-5b1a0d806617","Type":"ContainerStarted","Data":"fa94f26b5987ce74eeeb88bbcb0bb978d037b0b37753f82221c129307f7b32b6"} Jul 07 04:24:32.757080 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.757063 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/0.log" Jul 07 04:24:32.757179 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.757094 2563 generic.go:356] "Generic (PLEG): container finished" podID="a92b02e2-395b-44df-89ec-8d335b6e26fc" containerID="3f5f6a78d5d0686d0918431c50ed0210ee01711ffcf84f055dc326679a546a28" exitCode=255 Jul 07 04:24:32.757179 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.757161 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" event={"ID":"a92b02e2-395b-44df-89ec-8d335b6e26fc","Type":"ContainerDied","Data":"3f5f6a78d5d0686d0918431c50ed0210ee01711ffcf84f055dc326679a546a28"} Jul 07 04:24:32.757336 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.757322 2563 scope.go:117] "RemoveContainer" containerID="3f5f6a78d5d0686d0918431c50ed0210ee01711ffcf84f055dc326679a546a28" Jul 07 04:24:32.759193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.759159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkxhb" event={"ID":"8aac71c4-980e-4b85-a65f-e40c93af620c","Type":"ContainerStarted","Data":"89e89ae414578720c927dac32f8ac211d3acf0f2a468eb2936afe8a8abdffd87"} Jul 07 04:24:32.761262 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.761241 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" event={"ID":"a76e076d-8cd5-4e3c-9223-653a53e02f89","Type":"ContainerStarted","Data":"8da5ee8c219567d0afb289f718ab8d292dbe52a6de81695904fd2ea66a594340"} Jul 07 04:24:32.767408 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.767366 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-jnmnv" podStartSLOduration=25.125999308 podStartE2EDuration="33.76735267s" totalImagesPullingTime="8.641353362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:32.76679235 +0000 UTC m=+45.748726202" watchObservedRunningTime="2026-07-07 04:24:32.76735267 +0000 UTC m=+45.749286499" Jul 07 04:24:32.837354 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.836636 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" podStartSLOduration=28.137420522 podStartE2EDuration="36.836616716s" totalImagesPullingTime="8.699196194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:32.836330886 +0000 UTC m=+45.818264739" watchObservedRunningTime="2026-07-07 04:24:32.836616716 +0000 UTC m=+45.818550572" Jul 07 04:24:32.838106 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:32.838067 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pkxhb" podStartSLOduration=2.178093092 podStartE2EDuration="10.8380507s" totalImagesPullingTime="8.659957608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:32.811858888 +0000 UTC m=+45.793792741" watchObservedRunningTime="2026-07-07 04:24:32.8380507 +0000 UTC m=+45.819984554" Jul 07 04:24:33.027230 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.027172 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" podStartSLOduration=28.342834767 podStartE2EDuration="37.027148601s" totalImagesPullingTime="8.684313834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:32.876443735 +0000 UTC m=+45.858377587" watchObservedRunningTime="2026-07-07 04:24:33.027148601 +0000 UTC m=+46.009082455" Jul 07 04:24:33.028376 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.028352 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc"] Jul 07 04:24:33.039701 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.039679 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" Jul 07 04:24:33.042587 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.042216 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cg8c9\"" Jul 07 04:24:33.042587 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.042437 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:33.042587 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.042466 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 04:24:33.043053 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.042996 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc"] Jul 07 04:24:33.109388 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.109353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xs68\" (UniqueName: \"kubernetes.io/projected/bb4bc44d-0b6b-44db-8476-f04f39c5c9a4-kube-api-access-6xs68\") pod \"migrator-d57d58cc9-nrmlc\" (UID: \"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" Jul 07 04:24:33.210518 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.210489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs68\" (UniqueName: \"kubernetes.io/projected/bb4bc44d-0b6b-44db-8476-f04f39c5c9a4-kube-api-access-6xs68\") pod \"migrator-d57d58cc9-nrmlc\" (UID: \"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" Jul 07 04:24:33.222474 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.222441 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs68\" (UniqueName: \"kubernetes.io/projected/bb4bc44d-0b6b-44db-8476-f04f39c5c9a4-kube-api-access-6xs68\") pod \"migrator-d57d58cc9-nrmlc\" (UID: \"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" Jul 07 04:24:33.379345 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.379280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" Jul 07 04:24:33.512902 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.512846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:33.513074 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:33.512980 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:33.513074 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:33.513060 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:41.513041424 +0000 UTC m=+54.494975258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:33.528763 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.528697 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc"] Jul 07 04:24:33.530603 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:33.530577 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4bc44d_0b6b_44db_8476_f04f39c5c9a4.slice/crio-18bb3aff4c198cd253edd71429ca080aa265fa17cf3eb9a0d4c9223c8db3b1c2 WatchSource:0}: Error finding container 18bb3aff4c198cd253edd71429ca080aa265fa17cf3eb9a0d4c9223c8db3b1c2: Status 404 returned error can't find the container with id 18bb3aff4c198cd253edd71429ca080aa265fa17cf3eb9a0d4c9223c8db3b1c2 Jul 07 04:24:33.765379 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.765344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" event={"ID":"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4","Type":"ContainerStarted","Data":"18bb3aff4c198cd253edd71429ca080aa265fa17cf3eb9a0d4c9223c8db3b1c2"} Jul 07 04:24:33.768175 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.768154 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8436039-fc85-4333-88f1-d8ab9671e473" containerID="55d40b67b810f405872ee748b27a11bee3ef80a2e37466d413a8087988ed697b" exitCode=0 Jul 07 04:24:33.768280 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.768228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerDied","Data":"55d40b67b810f405872ee748b27a11bee3ef80a2e37466d413a8087988ed697b"} Jul 07 04:24:33.769801 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.769784 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:24:33.770161 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.770148 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/0.log" Jul 07 04:24:33.770217 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.770175 2563 generic.go:356] "Generic (PLEG): container finished" podID="a92b02e2-395b-44df-89ec-8d335b6e26fc" containerID="128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57" exitCode=255 Jul 07 04:24:33.770265 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.770240 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" event={"ID":"a92b02e2-395b-44df-89ec-8d335b6e26fc","Type":"ContainerDied","Data":"128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57"} Jul 07 04:24:33.770317 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.770272 2563 scope.go:117] "RemoveContainer" containerID="3f5f6a78d5d0686d0918431c50ed0210ee01711ffcf84f055dc326679a546a28" Jul 07 04:24:33.770448 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:33.770419 2563 scope.go:117] "RemoveContainer" containerID="128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57" Jul 07 04:24:33.770625 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:33.770580 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-cp2fz_openshift-console-operator(a92b02e2-395b-44df-89ec-8d335b6e26fc)\"" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" podUID="a92b02e2-395b-44df-89ec-8d335b6e26fc" Jul 07 04:24:34.776430 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:34.776395 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" event={"ID":"d8436039-fc85-4333-88f1-d8ab9671e473","Type":"ContainerStarted","Data":"40e24b593b97d35d80a5969a3e9957c62728eaf8d74325f6c356e45820d46d3b"} Jul 07 04:24:34.778037 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:34.778015 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:24:34.778346 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:34.778327 2563 scope.go:117] "RemoveContainer" containerID="128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57" Jul 07 04:24:34.778529 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:34.778508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-cp2fz_openshift-console-operator(a92b02e2-395b-44df-89ec-8d335b6e26fc)\"" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" podUID="a92b02e2-395b-44df-89ec-8d335b6e26fc" Jul 07 04:24:34.801481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:34.801438 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z8fdh" podStartSLOduration=10.861577876 podStartE2EDuration="46.801424699s" totalImagesPullingTime="35.939846823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:34.800098235 +0000 UTC m=+47.782032086" watchObservedRunningTime="2026-07-07 04:24:34.801424699 +0000 UTC m=+47.783358551" Jul 07 04:24:35.783003 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.782972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" event={"ID":"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4","Type":"ContainerStarted","Data":"ba652c996d998f52f4d4a8e25ab499d76fce9b422a8c5b8ad5b1ba0ec60a02ba"} Jul 07 04:24:35.783411 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.783022 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" event={"ID":"bb4bc44d-0b6b-44db-8476-f04f39c5c9a4","Type":"ContainerStarted","Data":"6bcd816ab50d6c27c4a893b165579c8de2931c8268bb0902b6d4e8020db682db"} Jul 07 04:24:35.800845 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.800803 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nrmlc" podStartSLOduration=0.916706099 podStartE2EDuration="2.800787681s" totalImagesPullingTime="1.884081582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:35.799946329 +0000 UTC m=+48.781880181" watchObservedRunningTime="2026-07-07 04:24:35.800787681 +0000 UTC m=+48.782721524" Jul 07 04:24:35.958570 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.958536 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x8bld"] Jul 07 04:24:35.979843 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.979817 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x8bld"] Jul 07 04:24:35.979959 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.979899 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:35.982555 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.982531 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 04:24:35.982674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.982559 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 04:24:35.982674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.982579 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 04:24:35.982674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.982582 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 04:24:35.982674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:35.982582 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vvfk8\"" Jul 07 04:24:36.032048 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.032027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca73433a-4804-4c05-986c-f785ca74547c-signing-key\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.032144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.032075 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca73433a-4804-4c05-986c-f785ca74547c-signing-cabundle\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.032197 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.032144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfsk\" (UniqueName: \"kubernetes.io/projected/ca73433a-4804-4c05-986c-f785ca74547c-kube-api-access-zlfsk\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.132996 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.132943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca73433a-4804-4c05-986c-f785ca74547c-signing-key\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.132996 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.132989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca73433a-4804-4c05-986c-f785ca74547c-signing-cabundle\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.133147 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.133121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfsk\" (UniqueName: \"kubernetes.io/projected/ca73433a-4804-4c05-986c-f785ca74547c-kube-api-access-zlfsk\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.133532 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.133513 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca73433a-4804-4c05-986c-f785ca74547c-signing-cabundle\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.135266 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.135247 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca73433a-4804-4c05-986c-f785ca74547c-signing-key\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.142974 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.142952 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfsk\" (UniqueName: \"kubernetes.io/projected/ca73433a-4804-4c05-986c-f785ca74547c-kube-api-access-zlfsk\") pod \"service-ca-758f599c5-x8bld\" (UID: \"ca73433a-4804-4c05-986c-f785ca74547c\") " pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.288749 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.288728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x8bld" Jul 07 04:24:36.433282 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.433008 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x8bld"] Jul 07 04:24:36.435343 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:36.435321 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca73433a_4804_4c05_986c_f785ca74547c.slice/crio-d74b39177a4c721f50bd064b4d89871157f5400f05a7920b9ced3c0f05d4cf43 WatchSource:0}: Error finding container d74b39177a4c721f50bd064b4d89871157f5400f05a7920b9ced3c0f05d4cf43: Status 404 returned error can't find the container with id d74b39177a4c721f50bd064b4d89871157f5400f05a7920b9ced3c0f05d4cf43 Jul 07 04:24:36.789490 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.789459 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x8bld" event={"ID":"ca73433a-4804-4c05-986c-f785ca74547c","Type":"ContainerStarted","Data":"d6e991c88aed9e7d1e88079626a55610803ffca1994bd2835a5f89e936e660de"} Jul 07 04:24:36.789891 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.789506 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x8bld" event={"ID":"ca73433a-4804-4c05-986c-f785ca74547c","Type":"ContainerStarted","Data":"d74b39177a4c721f50bd064b4d89871157f5400f05a7920b9ced3c0f05d4cf43"} Jul 07 04:24:36.818210 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:36.818164 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-x8bld" podStartSLOduration=1.818150493 podStartE2EDuration="1.818150493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:36.817319963 +0000 UTC m=+49.799253816" watchObservedRunningTime="2026-07-07 04:24:36.818150493 +0000 UTC m=+49.800084323" Jul 07 04:24:38.353422 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:38.353387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:38.353422 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:38.353428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:38.353953 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.353537 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 04:24:38.353953 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.353551 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9bf745f-8rbms: secret "image-registry-tls" not found Jul 07 04:24:38.353953 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.353598 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls podName:b2610943-f55b-47af-8683-928462bb9b38 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:54.35358262 +0000 UTC m=+67.335516450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls") pod "image-registry-6b9bf745f-8rbms" (UID: "b2610943-f55b-47af-8683-928462bb9b38") : secret "image-registry-tls" not found Jul 07 04:24:38.354110 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.353969 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 04:24:38.354110 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.354011 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls podName:729c14bc-b79b-4ddc-8a04-11a2fef542ab nodeName:}" failed. No retries permitted until 2026-07-07 04:24:54.353999952 +0000 UTC m=+67.335933782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vqdf4" (UID: "729c14bc-b79b-4ddc-8a04-11a2fef542ab") : secret "samples-operator-tls" not found Jul 07 04:24:38.454571 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:38.454534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:38.454735 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.454686 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 04:24:38.454811 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:38.454743 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls podName:1f3fc374-e44f-4652-9308-c1b1036a56cc nodeName:}" failed. No retries permitted until 2026-07-07 04:24:54.454726296 +0000 UTC m=+67.436660130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls") pod "dns-default-72lzq" (UID: "1f3fc374-e44f-4652-9308-c1b1036a56cc") : secret "dns-default-metrics-tls" not found Jul 07 04:24:41.578431 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:41.578397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:41.578963 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:41.578544 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 04:24:41.578963 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:41.578607 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert podName:092433f8-30e4-4c3c-919c-901d32b4a390 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:57.57859059 +0000 UTC m=+70.560524419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-npwxk" (UID: "092433f8-30e4-4c3c-919c-901d32b4a390") : secret "networking-console-plugin-cert" not found Jul 07 04:24:42.731970 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:42.731940 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:42.732325 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:42.732020 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:42.732325 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:42.732320 2563 scope.go:117] "RemoveContainer" containerID="128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57" Jul 07 04:24:42.732474 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:24:42.732452 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-cp2fz_openshift-console-operator(a92b02e2-395b-44df-89ec-8d335b6e26fc)\"" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" podUID="a92b02e2-395b-44df-89ec-8d335b6e26fc" Jul 07 04:24:47.701257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:47.701229 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhbbm" Jul 07 04:24:53.372333 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.372290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:53.372747 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.372397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:53.372747 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.372423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:53.375070 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.375050 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 04:24:53.375167 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.375068 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 04:24:53.375167 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.375129 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w5r\" (UniqueName: \"kubernetes.io/projected/71b154b2-7eca-4ccb-a50c-ab875f43daa0-kube-api-access-g5w5r\") pod \"network-check-target-bn4nk\" (UID: \"71b154b2-7eca-4ccb-a50c-ab875f43daa0\") " pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:53.385093 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.385072 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137a0240-63ec-41ac-910d-aafdbcfc59f0-metrics-certs\") pod \"network-metrics-daemon-7mhwl\" (UID: \"137a0240-63ec-41ac-910d-aafdbcfc59f0\") " pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:53.385211 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.385152 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca-original-pull-secret\") pod \"global-pull-secret-syncer-2tvqx\" (UID: \"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca\") " pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:53.439080 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.439063 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krb48\"" Jul 07 04:24:53.440902 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.440887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tvqx" Jul 07 04:24:53.447335 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.447320 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7mhwl" Jul 07 04:24:53.537193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.537168 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-krw7t\"" Jul 07 04:24:53.545073 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.544933 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:53.575971 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.575943 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tvqx"] Jul 07 04:24:53.578567 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:53.578534 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fffa7d6_6a77_4a16_97b1_8c0ee41b1cca.slice/crio-42bfbce1954c7775d860c658d8d1f7389c8668e44a61d54a5db4d172e6263a92 WatchSource:0}: Error finding container 42bfbce1954c7775d860c658d8d1f7389c8668e44a61d54a5db4d172e6263a92: Status 404 returned error can't find the container with id 42bfbce1954c7775d860c658d8d1f7389c8668e44a61d54a5db4d172e6263a92 Jul 07 04:24:53.604496 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.604433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7mhwl"] Jul 07 04:24:53.607076 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:53.607052 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137a0240_63ec_41ac_910d_aafdbcfc59f0.slice/crio-745aa218decd30c06feacba74679b0c482f37bd035ba37853b896a5900e7d04b WatchSource:0}: Error finding container 745aa218decd30c06feacba74679b0c482f37bd035ba37853b896a5900e7d04b: Status 404 returned error can't find the container with id 745aa218decd30c06feacba74679b0c482f37bd035ba37853b896a5900e7d04b Jul 07 04:24:53.675218 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.675188 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bn4nk"] Jul 07 04:24:53.677951 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:53.677927 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b154b2_7eca_4ccb_a50c_ab875f43daa0.slice/crio-77746b2056dfca676f60eded2a4ff8826f0128e372fc3ef68bf17256d67c437f WatchSource:0}: Error finding container 77746b2056dfca676f60eded2a4ff8826f0128e372fc3ef68bf17256d67c437f: Status 404 returned error can't find the container with id 77746b2056dfca676f60eded2a4ff8826f0128e372fc3ef68bf17256d67c437f Jul 07 04:24:53.841862 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.841830 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mhwl" event={"ID":"137a0240-63ec-41ac-910d-aafdbcfc59f0","Type":"ContainerStarted","Data":"745aa218decd30c06feacba74679b0c482f37bd035ba37853b896a5900e7d04b"} Jul 07 04:24:53.842903 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.842872 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tvqx" event={"ID":"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca","Type":"ContainerStarted","Data":"42bfbce1954c7775d860c658d8d1f7389c8668e44a61d54a5db4d172e6263a92"} Jul 07 04:24:53.844285 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.844264 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bn4nk" event={"ID":"71b154b2-7eca-4ccb-a50c-ab875f43daa0","Type":"ContainerStarted","Data":"0e4064fc81c9dc6ebb27ce66296e4ef39daa82607e6c4f8ec316aa5c067899d9"} Jul 07 04:24:53.844371 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.844302 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bn4nk" event={"ID":"71b154b2-7eca-4ccb-a50c-ab875f43daa0","Type":"ContainerStarted","Data":"77746b2056dfca676f60eded2a4ff8826f0128e372fc3ef68bf17256d67c437f"} Jul 07 04:24:53.844418 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.844383 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:24:53.859967 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:53.859926 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bn4nk" podStartSLOduration=65.85991431 podStartE2EDuration="1m5.85991431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:53.859298751 +0000 UTC m=+66.841232606" watchObservedRunningTime="2026-07-07 04:24:53.85991431 +0000 UTC m=+66.841848162" Jul 07 04:24:54.380787 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.380732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:54.381219 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.380810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:54.383690 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.383664 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/729c14bc-b79b-4ddc-8a04-11a2fef542ab-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vqdf4\" (UID: \"729c14bc-b79b-4ddc-8a04-11a2fef542ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:54.383879 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.383853 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"image-registry-6b9bf745f-8rbms\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:54.481971 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.481938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:54.484846 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.484819 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3fc374-e44f-4652-9308-c1b1036a56cc-metrics-tls\") pod \"dns-default-72lzq\" (UID: \"1f3fc374-e44f-4652-9308-c1b1036a56cc\") " pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:54.556840 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.556805 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6jvd\"" Jul 07 04:24:54.564642 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.564618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:54.570073 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.569986 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4k8g7\"" Jul 07 04:24:54.577959 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.577939 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" Jul 07 04:24:54.633822 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.633757 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qn6gg\"" Jul 07 04:24:54.640927 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.640898 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72lzq" Jul 07 04:24:54.870440 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:54.870416 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:24:54.873846 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:54.873814 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2610943_f55b_47af_8683_928462bb9b38.slice/crio-e8b25040f72814dd73f2689d2f86f685fa3ce965a4ddf479dfe5509ca3b1ad15 WatchSource:0}: Error finding container e8b25040f72814dd73f2689d2f86f685fa3ce965a4ddf479dfe5509ca3b1ad15: Status 404 returned error can't find the container with id e8b25040f72814dd73f2689d2f86f685fa3ce965a4ddf479dfe5509ca3b1ad15 Jul 07 04:24:55.085859 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.085829 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4"] Jul 07 04:24:55.089722 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.089708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72lzq"] Jul 07 04:24:55.093162 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:55.093134 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3fc374_e44f_4652_9308_c1b1036a56cc.slice/crio-804d9a7c38e9fc1c084d4ef11719a2b6bb00624a99ca146bb0b8e4c278668fea WatchSource:0}: Error finding container 804d9a7c38e9fc1c084d4ef11719a2b6bb00624a99ca146bb0b8e4c278668fea: Status 404 returned error can't find the container with id 804d9a7c38e9fc1c084d4ef11719a2b6bb00624a99ca146bb0b8e4c278668fea Jul 07 04:24:55.579476 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.579448 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-pswrc"] Jul 07 04:24:55.585190 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.585169 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57bbcb5b45-x2k47"] Jul 07 04:24:55.585308 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.585286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.587803 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.587759 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 04:24:55.587803 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.587797 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 04:24:55.588001 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.587851 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7cdw4\"" Jul 07 04:24:55.588001 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.587821 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 04:24:55.588168 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.588008 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb"] Jul 07 04:24:55.588168 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.588083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.588287 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.588275 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 04:24:55.590301 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.590279 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 04:24:55.590752 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.590733 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 04:24:55.590908 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.590885 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.591603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.591575 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 04:24:55.591702 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.591628 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 04:24:55.591754 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.591744 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 04:24:55.592010 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.591988 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 04:24:55.592147 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.592130 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wppzz\"" Jul 07 04:24:55.593426 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.593406 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 04:24:55.593514 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.593434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-pswrc"] Jul 07 04:24:55.593514 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.593507 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 04:24:55.593801 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.593757 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2rmmq\"" Jul 07 04:24:55.594850 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.594818 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 04:24:55.595201 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.595184 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 04:24:55.595297 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.595199 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 04:24:55.599895 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.599873 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb"] Jul 07 04:24:55.600840 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.600809 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57bbcb5b45-x2k47"] Jul 07 04:24:55.692013 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.691985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvnh\" (UniqueName: \"kubernetes.io/projected/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-kube-api-access-njvnh\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.692152 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.692152 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.692152 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkzk\" (UniqueName: \"kubernetes.io/projected/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-kube-api-access-qpkzk\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692155 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcmb\" (UniqueName: \"kubernetes.io/projected/d9049628-5100-4314-abaa-bb11f6760a4e-kube-api-access-spcmb\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-service-ca-bundle\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692224 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9049628-5100-4314-abaa-bb11f6760a4e-serving-cert\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692243 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-default-certificate\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-stats-auth\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.692709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-tmp\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-snapshots\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.692709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.692475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-metrics-certs\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.793783 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcmb\" (UniqueName: \"kubernetes.io/projected/d9049628-5100-4314-abaa-bb11f6760a4e-kube-api-access-spcmb\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.793910 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-service-ca-bundle\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.793910 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9049628-5100-4314-abaa-bb11f6760a4e-serving-cert\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.793910 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-default-certificate\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793937 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-stats-auth\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.793987 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-tmp\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-snapshots\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.794078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-metrics-certs\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.794376 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvnh\" (UniqueName: \"kubernetes.io/projected/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-kube-api-access-njvnh\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.794376 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.794376 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.794376 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.794199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkzk\" (UniqueName: \"kubernetes.io/projected/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-kube-api-access-qpkzk\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.795481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.795448 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-service-ca-bundle\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.796165 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.796137 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-tmp\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.797301 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.797218 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.797555 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.797480 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d9049628-5100-4314-abaa-bb11f6760a4e-snapshots\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.797903 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.797838 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9049628-5100-4314-abaa-bb11f6760a4e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.798019 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.797939 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.799864 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.799825 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-stats-auth\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.799958 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.799901 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-metrics-certs\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.801222 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.801163 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-default-certificate\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.801222 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.801166 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d9049628-5100-4314-abaa-bb11f6760a4e-serving-cert\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.801615 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.801578 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.814181 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.814130 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkzk\" (UniqueName: \"kubernetes.io/projected/ecd39bb2-bc3b-481f-97fd-96ee13e13f27-kube-api-access-qpkzk\") pod \"cluster-monitoring-operator-64786d9cbb-f8zbb\" (UID: \"ecd39bb2-bc3b-481f-97fd-96ee13e13f27\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:55.815955 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.815927 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvnh\" (UniqueName: \"kubernetes.io/projected/15545f1f-bb57-4d18-8c65-cdf59a7a5dc9-kube-api-access-njvnh\") pod \"router-default-57bbcb5b45-x2k47\" (UID: \"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9\") " pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.816380 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.816335 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcmb\" (UniqueName: \"kubernetes.io/projected/d9049628-5100-4314-abaa-bb11f6760a4e-kube-api-access-spcmb\") pod \"insights-operator-5c847dcfc4-pswrc\" (UID: \"d9049628-5100-4314-abaa-bb11f6760a4e\") " pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.856438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.856366 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" event={"ID":"b2610943-f55b-47af-8683-928462bb9b38","Type":"ContainerStarted","Data":"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e"} Jul 07 04:24:55.856438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.856429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" event={"ID":"b2610943-f55b-47af-8683-928462bb9b38","Type":"ContainerStarted","Data":"e8b25040f72814dd73f2689d2f86f685fa3ce965a4ddf479dfe5509ca3b1ad15"} Jul 07 04:24:55.857183 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.857164 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:24:55.858856 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.858827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72lzq" event={"ID":"1f3fc374-e44f-4652-9308-c1b1036a56cc","Type":"ContainerStarted","Data":"804d9a7c38e9fc1c084d4ef11719a2b6bb00624a99ca146bb0b8e4c278668fea"} Jul 07 04:24:55.862198 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.861543 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mhwl" event={"ID":"137a0240-63ec-41ac-910d-aafdbcfc59f0","Type":"ContainerStarted","Data":"6bcc68824dcfd0fe25314d814cd6c5c0f494c30c204a2de23ebed4b8b059d70f"} Jul 07 04:24:55.862198 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.861589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7mhwl" event={"ID":"137a0240-63ec-41ac-910d-aafdbcfc59f0","Type":"ContainerStarted","Data":"99ee79615d1cf969152f2abd19c5ab770b8c3dba0c289b1c981146e3ece6f33f"} Jul 07 04:24:55.863317 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.863290 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" event={"ID":"729c14bc-b79b-4ddc-8a04-11a2fef542ab","Type":"ContainerStarted","Data":"6e8a4305121e3983cc9f228abda2b20ad8c4c258dd18625ef3b5757ac4ef8e6a"} Jul 07 04:24:55.883709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.883501 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" podStartSLOduration=67.883482947 podStartE2EDuration="1m7.883482947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:55.881214602 +0000 UTC m=+68.863148456" watchObservedRunningTime="2026-07-07 04:24:55.883482947 +0000 UTC m=+68.865416801" Jul 07 04:24:55.897857 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.897833 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" Jul 07 04:24:55.900384 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.899737 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7mhwl" podStartSLOduration=66.75034771 podStartE2EDuration="1m7.899723488s" totalImagesPullingTime="1.149375778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:55.899203793 +0000 UTC m=+68.881137647" watchObservedRunningTime="2026-07-07 04:24:55.899723488 +0000 UTC m=+68.881657337" Jul 07 04:24:55.906145 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.906121 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:55.910606 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:55.910446 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" Jul 07 04:24:56.123844 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.123752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57bbcb5b45-x2k47"] Jul 07 04:24:56.127177 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.127152 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-pswrc"] Jul 07 04:24:56.127611 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:56.127577 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15545f1f_bb57_4d18_8c65_cdf59a7a5dc9.slice/crio-ef645f02476f17a96c0db916a317abd40c1e1217f42a98092804e0dbf18ef993 WatchSource:0}: Error finding container ef645f02476f17a96c0db916a317abd40c1e1217f42a98092804e0dbf18ef993: Status 404 returned error can't find the container with id ef645f02476f17a96c0db916a317abd40c1e1217f42a98092804e0dbf18ef993 Jul 07 04:24:56.130829 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:56.130804 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9049628_5100_4314_abaa_bb11f6760a4e.slice/crio-6262860441d2196be90d18ca03118bb94b2bb45ac3349e8fa9931172738b7e14 WatchSource:0}: Error finding container 6262860441d2196be90d18ca03118bb94b2bb45ac3349e8fa9931172738b7e14: Status 404 returned error can't find the container with id 6262860441d2196be90d18ca03118bb94b2bb45ac3349e8fa9931172738b7e14 Jul 07 04:24:56.146478 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.146450 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb"] Jul 07 04:24:56.151888 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:24:56.151863 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd39bb2_bc3b_481f_97fd_96ee13e13f27.slice/crio-29d4ce8600b8149723454603efe5ddb8a6ea4462252ade1a035f75400e11115b WatchSource:0}: Error finding container 29d4ce8600b8149723454603efe5ddb8a6ea4462252ade1a035f75400e11115b: Status 404 returned error can't find the container with id 29d4ce8600b8149723454603efe5ddb8a6ea4462252ade1a035f75400e11115b Jul 07 04:24:56.522210 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.522182 2563 scope.go:117] "RemoveContainer" containerID="128730f1866ba282240a8688cc6d516eeab5ddfdeb8447924651b5c1e52f2a57" Jul 07 04:24:56.879508 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.879412 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" event={"ID":"ecd39bb2-bc3b-481f-97fd-96ee13e13f27","Type":"ContainerStarted","Data":"29d4ce8600b8149723454603efe5ddb8a6ea4462252ade1a035f75400e11115b"} Jul 07 04:24:56.881974 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.881629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" event={"ID":"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9","Type":"ContainerStarted","Data":"beb8ae368ee2be762cda800955c68dfec5cffa0cb96ad5a49af4f6151644d354"} Jul 07 04:24:56.881974 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.881684 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" event={"ID":"15545f1f-bb57-4d18-8c65-cdf59a7a5dc9","Type":"ContainerStarted","Data":"ef645f02476f17a96c0db916a317abd40c1e1217f42a98092804e0dbf18ef993"} Jul 07 04:24:56.886042 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.886005 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:24:56.886165 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.886116 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" event={"ID":"a92b02e2-395b-44df-89ec-8d335b6e26fc","Type":"ContainerStarted","Data":"55475a5066101c2700f6fabaa039cb2b9a857c5ade953ea4f372c724027e8410"} Jul 07 04:24:56.886730 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.886708 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:56.893492 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.893247 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" Jul 07 04:24:56.898229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.898201 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" event={"ID":"d9049628-5100-4314-abaa-bb11f6760a4e","Type":"ContainerStarted","Data":"6262860441d2196be90d18ca03118bb94b2bb45ac3349e8fa9931172738b7e14"} Jul 07 04:24:56.907497 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.906557 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:56.909544 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.909501 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:56.921468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.921421 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-cp2fz" podStartSLOduration=54.140412138 podStartE2EDuration="1m2.921402483s" totalImagesPullingTime="8.780990345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:56.920329231 +0000 UTC m=+69.902263083" watchObservedRunningTime="2026-07-07 04:24:56.921402483 +0000 UTC m=+69.903336336" Jul 07 04:24:56.921991 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:56.921954 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" podStartSLOduration=1.921941349 podStartE2EDuration="1.921941349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:56.901726599 +0000 UTC m=+69.883660451" watchObservedRunningTime="2026-07-07 04:24:56.921941349 +0000 UTC m=+69.903875204" Jul 07 04:24:57.612125 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.612084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:57.615002 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.614949 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/092433f8-30e4-4c3c-919c-901d32b4a390-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-npwxk\" (UID: \"092433f8-30e4-4c3c-919c-901d32b4a390\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:57.647482 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.647459 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k5rdd\"" Jul 07 04:24:57.655053 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.655032 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" Jul 07 04:24:57.903546 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.903464 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:24:57.905012 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:24:57.904986 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57bbcb5b45-x2k47" Jul 07 04:25:01.623389 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.623325 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-npwxk"] Jul 07 04:25:01.638342 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:25:01.638303 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092433f8_30e4_4c3c_919c_901d32b4a390.slice/crio-a3c607f0ead1e53a4675b3dd7add5b1198125f4eacddcdb932022d786481151e WatchSource:0}: Error finding container a3c607f0ead1e53a4675b3dd7add5b1198125f4eacddcdb932022d786481151e: Status 404 returned error can't find the container with id a3c607f0ead1e53a4675b3dd7add5b1198125f4eacddcdb932022d786481151e Jul 07 04:25:01.919383 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.919345 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" event={"ID":"092433f8-30e4-4c3c-919c-901d32b4a390","Type":"ContainerStarted","Data":"a3c607f0ead1e53a4675b3dd7add5b1198125f4eacddcdb932022d786481151e"} Jul 07 04:25:01.921334 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.921301 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" event={"ID":"729c14bc-b79b-4ddc-8a04-11a2fef542ab","Type":"ContainerStarted","Data":"fe80a02779d57d0413de62b9bb7ca86ab141a2ab426172a844bf0e7a92c65ff0"} Jul 07 04:25:01.921558 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.921361 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" event={"ID":"729c14bc-b79b-4ddc-8a04-11a2fef542ab","Type":"ContainerStarted","Data":"5b73327ec3e0c8fb7f26d8700832963f8f349c68113097eb859ee831cabe1af2"} Jul 07 04:25:01.923062 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.923036 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" event={"ID":"d9049628-5100-4314-abaa-bb11f6760a4e","Type":"ContainerStarted","Data":"85d8b8b91731eb351d5a9a37b04957d9639e441b6b6406f6b4ce2a26b8d59b79"} Jul 07 04:25:01.924667 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.924641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tvqx" event={"ID":"8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca","Type":"ContainerStarted","Data":"001b4a1fbcc55f7d29097a4c9af45f5dd0adbfb5507a4004921b9e686d30d763"} Jul 07 04:25:01.926723 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.926702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72lzq" event={"ID":"1f3fc374-e44f-4652-9308-c1b1036a56cc","Type":"ContainerStarted","Data":"242492c11bb4b982d271cb6029b6d9e2dcde6731d25bd36c0ace1abaf9426841"} Jul 07 04:25:01.926854 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.926746 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72lzq" event={"ID":"1f3fc374-e44f-4652-9308-c1b1036a56cc","Type":"ContainerStarted","Data":"de1997e72090af9c9b636aed472d2d5b57020aed414d39a10f741e67eef204d0"} Jul 07 04:25:01.926854 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.926820 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-72lzq" Jul 07 04:25:01.928257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.928239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" event={"ID":"ecd39bb2-bc3b-481f-97fd-96ee13e13f27","Type":"ContainerStarted","Data":"9b1e10dc8c205b5b032c2b6041f049dd5cc02ba6aaa83c00056e0b05d3a610b6"} Jul 07 04:25:01.938934 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.938882 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vqdf4" podStartSLOduration=63.602562553 podStartE2EDuration="1m9.938862879s" totalImagesPullingTime="6.336300326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:01.937598456 +0000 UTC m=+74.919532308" watchObservedRunningTime="2026-07-07 04:25:01.938862879 +0000 UTC m=+74.920796729" Jul 07 04:25:01.952747 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.952681 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-pswrc" podStartSLOduration=1.6023818250000001 podStartE2EDuration="6.952662669s" totalImagesPullingTime="5.350280844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:01.95222748 +0000 UTC m=+74.934161335" watchObservedRunningTime="2026-07-07 04:25:01.952662669 +0000 UTC m=+74.934596522" Jul 07 04:25:01.967410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.967348 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2tvqx" podStartSLOduration=66.068226977 podStartE2EDuration="1m13.967332151s" totalImagesPullingTime="7.899105174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:01.966268838 +0000 UTC m=+74.948202690" watchObservedRunningTime="2026-07-07 04:25:01.967332151 +0000 UTC m=+74.949265981" Jul 07 04:25:01.987389 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:01.987340 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-72lzq" podStartSLOduration=33.669423614 podStartE2EDuration="39.987324526s" totalImagesPullingTime="6.317900912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:01.986232292 +0000 UTC m=+74.968166156" watchObservedRunningTime="2026-07-07 04:25:01.987324526 +0000 UTC m=+74.969258717" Jul 07 04:25:02.242273 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.242219 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-f8zbb" podStartSLOduration=1.9166600809999998 podStartE2EDuration="7.242198867s" totalImagesPullingTime="5.325538786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:02.013557615 +0000 UTC m=+74.995491467" watchObservedRunningTime="2026-07-07 04:25:02.242198867 +0000 UTC m=+75.224132756" Jul 07 04:25:02.242421 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.242349 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tmlwm"] Jul 07 04:25:02.266085 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.266056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tmlwm"] Jul 07 04:25:02.266238 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.266169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.268359 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.268335 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 04:25:02.268492 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.268337 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 04:25:02.268492 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.268487 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwn6d\"" Jul 07 04:25:02.457616 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.457569 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49ba2da0-f024-44f3-a481-4a9ce844b390-crio-socket\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.457814 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.457624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.457814 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.457742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49ba2da0-f024-44f3-a481-4a9ce844b390-data-volume\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.457814 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.457806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.457978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.457874 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m4f\" (UniqueName: \"kubernetes.io/projected/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-api-access-t8m4f\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558820 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m4f\" (UniqueName: \"kubernetes.io/projected/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-api-access-t8m4f\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558820 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49ba2da0-f024-44f3-a481-4a9ce844b390-crio-socket\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558989 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558989 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49ba2da0-f024-44f3-a481-4a9ce844b390-data-volume\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558989 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558989 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.558927 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49ba2da0-f024-44f3-a481-4a9ce844b390-crio-socket\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.558989 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:02.558949 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:02.559141 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:02.559006 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:02.559141 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:02.559023 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:03.059001405 +0000 UTC m=+76.040935240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : configmap "kube-rbac-proxy" not found Jul 07 04:25:02.559141 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:02.559042 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:03.05903157 +0000 UTC m=+76.040965405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:02.559244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.559184 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49ba2da0-f024-44f3-a481-4a9ce844b390-data-volume\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.571195 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.571167 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m4f\" (UniqueName: \"kubernetes.io/projected/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-api-access-t8m4f\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:02.924247 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:02.924161 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72lzq_1f3fc374-e44f-4652-9308-c1b1036a56cc/dns/0.log" Jul 07 04:25:03.063100 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:03.063062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:03.063291 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:03.063152 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:03.063291 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:03.063215 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:04.063197166 +0000 UTC m=+77.045131018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : configmap "kube-rbac-proxy" not found Jul 07 04:25:03.063458 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:03.063434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:03.063546 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:03.063515 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:03.063615 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:03.063584 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:04.063566365 +0000 UTC m=+77.045500197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:03.124279 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:03.124253 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72lzq_1f3fc374-e44f-4652-9308-c1b1036a56cc/kube-rbac-proxy/0.log" Jul 07 04:25:03.938146 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:03.938111 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" event={"ID":"092433f8-30e4-4c3c-919c-901d32b4a390","Type":"ContainerStarted","Data":"b9e490672aa592cc9fad324f0904a25dea21da36d55a3cf09705e77e9d03bb80"} Jul 07 04:25:03.954557 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:03.954506 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-npwxk" podStartSLOduration=37.141909521 podStartE2EDuration="38.954489588s" totalImagesPullingTime="1.812580067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:03.953868576 +0000 UTC m=+76.935802428" watchObservedRunningTime="2026-07-07 04:25:03.954489588 +0000 UTC m=+76.936423441" Jul 07 04:25:04.073127 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:04.073085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:04.073307 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:04.073219 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:04.073307 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:04.073241 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:04.073307 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:04.073303 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:06.073286757 +0000 UTC m=+79.055220587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:04.073432 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:04.073321 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:04.073432 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:04.073370 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:06.073357896 +0000 UTC m=+79.055291726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : configmap "kube-rbac-proxy" not found Jul 07 04:25:04.324460 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:04.324430 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvmr2_e58e6ac7-8ee6-4790-8625-54d10183203e/dns-node-resolver/0.log" Jul 07 04:25:04.724834 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:04.724735 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b9bf745f-8rbms_b2610943-f55b-47af-8683-928462bb9b38/registry/0.log" Jul 07 04:25:05.524422 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:05.524393 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c8tf5_42292ea1-1500-4541-9321-06324275ce72/node-ca/0.log" Jul 07 04:25:05.724977 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:05.724948 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bbcb5b45-x2k47_15545f1f-bb57-4d18-8c65-cdf59a7a5dc9/router/0.log" Jul 07 04:25:06.088968 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:06.088929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:06.089164 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:06.088992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:06.089164 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:06.089065 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:06.089164 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:06.089124 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:06.089164 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:06.089130 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:10.089114569 +0000 UTC m=+83.071048398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : configmap "kube-rbac-proxy" not found Jul 07 04:25:06.089302 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:06.089178 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls podName:49ba2da0-f024-44f3-a481-4a9ce844b390 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:10.089165642 +0000 UTC m=+83.071099471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tmlwm" (UID: "49ba2da0-f024-44f3-a481-4a9ce844b390") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:06.324354 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:06.324320 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkxhb_8aac71c4-980e-4b85-a65f-e40c93af620c/serve-healthcheck-canary/0.log" Jul 07 04:25:10.115357 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.115308 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:10.115728 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.115386 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:10.115886 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.115869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49ba2da0-f024-44f3-a481-4a9ce844b390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:10.117615 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.117593 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49ba2da0-f024-44f3-a481-4a9ce844b390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tmlwm\" (UID: \"49ba2da0-f024-44f3-a481-4a9ce844b390\") " pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:10.374842 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.374758 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tmlwm" Jul 07 04:25:10.504843 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.504819 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tmlwm"] Jul 07 04:25:10.507352 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:25:10.507327 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ba2da0_f024_44f3_a481_4a9ce844b390.slice/crio-2338b95809e5b42c2800adba6a2ff39d3ecd61c44cad4a69d87954101a518365 WatchSource:0}: Error finding container 2338b95809e5b42c2800adba6a2ff39d3ecd61c44cad4a69d87954101a518365: Status 404 returned error can't find the container with id 2338b95809e5b42c2800adba6a2ff39d3ecd61c44cad4a69d87954101a518365 Jul 07 04:25:10.963661 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:10.963627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tmlwm" event={"ID":"49ba2da0-f024-44f3-a481-4a9ce844b390","Type":"ContainerStarted","Data":"2338b95809e5b42c2800adba6a2ff39d3ecd61c44cad4a69d87954101a518365"} Jul 07 04:25:11.935440 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:11.935407 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-72lzq" Jul 07 04:25:14.569392 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:14.569362 2563 patch_prober.go:28] interesting pod/image-registry-6b9bf745f-8rbms container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 04:25:14.569674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:14.569403 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" podUID="b2610943-f55b-47af-8683-928462bb9b38" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 04:25:14.982494 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:14.982451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tmlwm" event={"ID":"49ba2da0-f024-44f3-a481-4a9ce844b390","Type":"ContainerStarted","Data":"4f8363c5a5a91cb5d27a277e4efa7ece4d363a04b4a90e7601c24634e104a97a"} Jul 07 04:25:15.989186 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:15.989144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tmlwm" event={"ID":"49ba2da0-f024-44f3-a481-4a9ce844b390","Type":"ContainerStarted","Data":"08d37ad20b4c641db11907d64d381b927000924631854855b9330894c384fd71"} Jul 07 04:25:17.907886 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:17.907858 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:25:18.521229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:18.521198 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:25:19.002954 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:19.002922 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tmlwm" event={"ID":"49ba2da0-f024-44f3-a481-4a9ce844b390","Type":"ContainerStarted","Data":"97b4c3c7883fac4bbcf7967250b0e21479f70dcb4cabde8609deb96e559fe90b"} Jul 07 04:25:19.031338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:19.031293 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tmlwm" podStartSLOduration=9.489965162 podStartE2EDuration="17.031279229s" totalImagesPullingTime="7.541314067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:25:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:19.030439308 +0000 UTC m=+92.012373164" watchObservedRunningTime="2026-07-07 04:25:19.031279229 +0000 UTC m=+92.013213059" Jul 07 04:25:21.539825 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.539791 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ngv2f"] Jul 07 04:25:21.543256 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.543236 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.545922 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.545783 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 04:25:21.545922 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.545804 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 04:25:21.545922 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.545861 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 04:25:21.546882 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.546864 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8825k\"" Jul 07 04:25:21.546976 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.546887 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 04:25:21.606853 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.606827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.606981 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.606871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-textfile\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.606981 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.606923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.606981 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.606963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.607144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.606990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-root\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.607144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.607021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mlm\" (UniqueName: \"kubernetes.io/projected/16682724-bbc5-423f-aff3-79b1865d19bf-kube-api-access-k2mlm\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.607144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.607045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-metrics-client-ca\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.607144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.607090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-sys\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.607144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.607118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-wtmp\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708050 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mlm\" (UniqueName: \"kubernetes.io/projected/16682724-bbc5-423f-aff3-79b1865d19bf-kube-api-access-k2mlm\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708220 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-metrics-client-ca\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708307 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-sys\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708366 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-wtmp\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708366 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708351 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-sys\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708470 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708470 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708419 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-textfile\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708470 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-root\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708530 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-wtmp\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708587 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16682724-bbc5-423f-aff3-79b1865d19bf-root\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708830 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-textfile\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708830 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:21.708823 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 04:25:21.708930 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.708865 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-metrics-client-ca\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.708930 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:21.708873 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls podName:16682724-bbc5-423f-aff3-79b1865d19bf nodeName:}" failed. No retries permitted until 2026-07-07 04:25:22.208856307 +0000 UTC m=+95.190790146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls") pod "node-exporter-ngv2f" (UID: "16682724-bbc5-423f-aff3-79b1865d19bf") : secret "node-exporter-tls" not found Jul 07 04:25:21.709126 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.709106 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.710940 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.710919 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:21.718161 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:21.718140 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mlm\" (UniqueName: \"kubernetes.io/projected/16682724-bbc5-423f-aff3-79b1865d19bf-kube-api-access-k2mlm\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:22.211851 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:22.211811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:22.214214 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:22.214182 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16682724-bbc5-423f-aff3-79b1865d19bf-node-exporter-tls\") pod \"node-exporter-ngv2f\" (UID: \"16682724-bbc5-423f-aff3-79b1865d19bf\") " pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:22.453116 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:22.453083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngv2f" Jul 07 04:25:22.471990 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:25:22.471943 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16682724_bbc5_423f_aff3_79b1865d19bf.slice/crio-2f6517415bd270ed8afc458cdba7dba410438fd8f3b92d81243afe3e6621b5e7 WatchSource:0}: Error finding container 2f6517415bd270ed8afc458cdba7dba410438fd8f3b92d81243afe3e6621b5e7: Status 404 returned error can't find the container with id 2f6517415bd270ed8afc458cdba7dba410438fd8f3b92d81243afe3e6621b5e7 Jul 07 04:25:23.019847 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:23.019813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv2f" event={"ID":"16682724-bbc5-423f-aff3-79b1865d19bf","Type":"ContainerStarted","Data":"2f6517415bd270ed8afc458cdba7dba410438fd8f3b92d81243afe3e6621b5e7"} Jul 07 04:25:24.025569 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:24.025532 2563 generic.go:356] "Generic (PLEG): container finished" podID="16682724-bbc5-423f-aff3-79b1865d19bf" containerID="cda725c28ca4670cd546c0cd9cfdab7155c6d4156a71fea8c9a14f19339c8348" exitCode=0 Jul 07 04:25:24.025937 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:24.025612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv2f" event={"ID":"16682724-bbc5-423f-aff3-79b1865d19bf","Type":"ContainerDied","Data":"cda725c28ca4670cd546c0cd9cfdab7155c6d4156a71fea8c9a14f19339c8348"} Jul 07 04:25:24.851141 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:24.851113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bn4nk" Jul 07 04:25:25.035588 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:25.035547 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv2f" event={"ID":"16682724-bbc5-423f-aff3-79b1865d19bf","Type":"ContainerStarted","Data":"dd0aa5ce19961ffcf1f336d231cfd160bee7eadd01298ee2896d28c0dc8d18a0"} Jul 07 04:25:25.036009 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:25.035617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv2f" event={"ID":"16682724-bbc5-423f-aff3-79b1865d19bf","Type":"ContainerStarted","Data":"b5ba6c4dc988d53bb3c01f85004cb42a0cfabc333d204842ad3c14cb4f71f516"} Jul 07 04:25:25.053940 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:25.053888 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ngv2f" podStartSLOduration=2.8950953139999998 podStartE2EDuration="4.053873359s" totalImagesPullingTime="1.096911843s" totalInitContainerRuntime="61.866202ms" isStatefulPod=true podCreationTimestamp="2026-07-07 04:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:25.052101923 +0000 UTC m=+98.034035792" watchObservedRunningTime="2026-07-07 04:25:25.053873359 +0000 UTC m=+98.035807210" Jul 07 04:25:38.084862 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:38.084829 2563 generic.go:356] "Generic (PLEG): container finished" podID="2c61aea7-54b3-4024-8f7d-5b1a0d806617" containerID="fa94f26b5987ce74eeeb88bbcb0bb978d037b0b37753f82221c129307f7b32b6" exitCode=0 Jul 07 04:25:38.085245 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:38.084872 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" event={"ID":"2c61aea7-54b3-4024-8f7d-5b1a0d806617","Type":"ContainerDied","Data":"fa94f26b5987ce74eeeb88bbcb0bb978d037b0b37753f82221c129307f7b32b6"} Jul 07 04:25:38.085245 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:38.085185 2563 scope.go:117] "RemoveContainer" containerID="fa94f26b5987ce74eeeb88bbcb0bb978d037b0b37753f82221c129307f7b32b6" Jul 07 04:25:39.090309 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:39.090274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wnvlz" event={"ID":"2c61aea7-54b3-4024-8f7d-5b1a0d806617","Type":"ContainerStarted","Data":"8eb0ad22e438c9c2f998ff289a09cf623f8500a49324ff1b7660c7cca4d9d037"} Jul 07 04:25:43.541354 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.541312 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" podUID="b2610943-f55b-47af-8683-928462bb9b38" containerName="registry" containerID="cri-o://4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e" gracePeriod=30 Jul 07 04:25:43.785574 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.785552 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:25:43.856566 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856510 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856566 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856543 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856571 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856603 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856631 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856648 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856669 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.856718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.856703 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ts7j\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j\") pod \"b2610943-f55b-47af-8683-928462bb9b38\" (UID: \"b2610943-f55b-47af-8683-928462bb9b38\") " Jul 07 04:25:43.857258 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.857223 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:43.857608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.857540 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:43.859794 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.859657 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:25:43.862381 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.861847 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:25:43.863996 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.863931 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:25:43.864896 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.864871 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:25:43.864987 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.864956 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j" (OuterVolumeSpecName: "kube-api-access-7ts7j") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "kube-api-access-7ts7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:25:43.868399 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.868374 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b2610943-f55b-47af-8683-928462bb9b38" (UID: "b2610943-f55b-47af-8683-928462bb9b38"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:25:43.957571 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957551 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-registry-certificates\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957571 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957572 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2610943-f55b-47af-8683-928462bb9b38-ca-trust-extracted\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957581 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-installation-pull-secrets\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957590 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2610943-f55b-47af-8683-928462bb9b38-image-registry-private-configuration\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957600 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-registry-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957608 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-bound-sa-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957616 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2610943-f55b-47af-8683-928462bb9b38-trusted-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:43.957677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:43.957623 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ts7j\" (UniqueName: \"kubernetes.io/projected/b2610943-f55b-47af-8683-928462bb9b38-kube-api-access-7ts7j\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:25:44.110113 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.110034 2563 generic.go:356] "Generic (PLEG): container finished" podID="b2610943-f55b-47af-8683-928462bb9b38" containerID="4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e" exitCode=0 Jul 07 04:25:44.110113 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.110097 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" Jul 07 04:25:44.110245 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.110098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" event={"ID":"b2610943-f55b-47af-8683-928462bb9b38","Type":"ContainerDied","Data":"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e"} Jul 07 04:25:44.110245 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.110239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9bf745f-8rbms" event={"ID":"b2610943-f55b-47af-8683-928462bb9b38","Type":"ContainerDied","Data":"e8b25040f72814dd73f2689d2f86f685fa3ce965a4ddf479dfe5509ca3b1ad15"} Jul 07 04:25:44.110353 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.110262 2563 scope.go:117] "RemoveContainer" containerID="4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e" Jul 07 04:25:44.118503 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.118485 2563 scope.go:117] "RemoveContainer" containerID="4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e" Jul 07 04:25:44.118786 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:25:44.118740 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e\": container with ID starting with 4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e not found: ID does not exist" containerID="4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e" Jul 07 04:25:44.118854 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.118792 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e"} err="failed to get container status \"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e\": rpc error: code = NotFound desc = could not find container \"4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e\": container with ID starting with 4c07e359c29f48bdef5d81c1e0883409199f2a6ea9b7b7409e97324640995e1e not found: ID does not exist" Jul 07 04:25:44.133249 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.133230 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:25:44.140960 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:44.140940 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b9bf745f-8rbms"] Jul 07 04:25:45.526511 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:25:45.526477 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2610943-f55b-47af-8683-928462bb9b38" path="/var/lib/kubelet/pods/b2610943-f55b-47af-8683-928462bb9b38/volumes" Jul 07 04:26:03.186044 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:26:03.186010 2563 generic.go:356] "Generic (PLEG): container finished" podID="a76e076d-8cd5-4e3c-9223-653a53e02f89" containerID="8da5ee8c219567d0afb289f718ab8d292dbe52a6de81695904fd2ea66a594340" exitCode=0 Jul 07 04:26:03.186396 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:26:03.186084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" event={"ID":"a76e076d-8cd5-4e3c-9223-653a53e02f89","Type":"ContainerDied","Data":"8da5ee8c219567d0afb289f718ab8d292dbe52a6de81695904fd2ea66a594340"} Jul 07 04:26:03.186396 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:26:03.186393 2563 scope.go:117] "RemoveContainer" containerID="8da5ee8c219567d0afb289f718ab8d292dbe52a6de81695904fd2ea66a594340" Jul 07 04:26:04.191476 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:26:04.191438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-svjfs" event={"ID":"a76e076d-8cd5-4e3c-9223-653a53e02f89","Type":"ContainerStarted","Data":"cf15304e7124fbe9daebabbd040ca382a79d8fff335c646a44b50481ea826b9b"} Jul 07 04:28:05.346296 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.346260 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2"] Jul 07 04:28:05.346807 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.346536 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2610943-f55b-47af-8683-928462bb9b38" containerName="registry" Jul 07 04:28:05.346807 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.346546 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2610943-f55b-47af-8683-928462bb9b38" containerName="registry" Jul 07 04:28:05.346807 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.346599 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2610943-f55b-47af-8683-928462bb9b38" containerName="registry" Jul 07 04:28:05.349552 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.349531 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.352061 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.352038 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:28:05.352184 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.352040 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vrc2g\"" Jul 07 04:28:05.352184 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.352073 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 04:28:05.362381 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.362363 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2"] Jul 07 04:28:05.429086 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.429057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24vr\" (UniqueName: \"kubernetes.io/projected/2eae20a5-5aec-4e63-9301-46c86433732a-kube-api-access-w24vr\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.429193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.429115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae20a5-5aec-4e63-9301-46c86433732a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.529910 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.529881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24vr\" (UniqueName: \"kubernetes.io/projected/2eae20a5-5aec-4e63-9301-46c86433732a-kube-api-access-w24vr\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.530046 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.529977 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae20a5-5aec-4e63-9301-46c86433732a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.530338 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.530318 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae20a5-5aec-4e63-9301-46c86433732a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.540630 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.540603 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24vr\" (UniqueName: \"kubernetes.io/projected/2eae20a5-5aec-4e63-9301-46c86433732a-kube-api-access-w24vr\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rqp2\" (UID: \"2eae20a5-5aec-4e63-9301-46c86433732a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.658718 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.658667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" Jul 07 04:28:05.790243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:05.790216 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2"] Jul 07 04:28:05.793278 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:28:05.793253 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eae20a5_5aec_4e63_9301_46c86433732a.slice/crio-f8e32bb75b8574fc42828377c502d4304c2894ddbd2af12e1866975c06e2a737 WatchSource:0}: Error finding container f8e32bb75b8574fc42828377c502d4304c2894ddbd2af12e1866975c06e2a737: Status 404 returned error can't find the container with id f8e32bb75b8574fc42828377c502d4304c2894ddbd2af12e1866975c06e2a737 Jul 07 04:28:06.640723 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:06.640686 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" event={"ID":"2eae20a5-5aec-4e63-9301-46c86433732a","Type":"ContainerStarted","Data":"f8e32bb75b8574fc42828377c502d4304c2894ddbd2af12e1866975c06e2a737"} Jul 07 04:28:08.650579 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:08.650500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" event={"ID":"2eae20a5-5aec-4e63-9301-46c86433732a","Type":"ContainerStarted","Data":"fade93c3b754d092e1740083cb03ddde274bd56f40353568a8a4144cd999287b"} Jul 07 04:28:08.673231 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:08.673185 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rqp2" podStartSLOduration=1.118115566 podStartE2EDuration="3.673169236s" totalImagesPullingTime="2.55505367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:08.671728033 +0000 UTC m=+261.653661886" watchObservedRunningTime="2026-07-07 04:28:08.673169236 +0000 UTC m=+261.655103088" Jul 07 04:28:09.101502 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.101468 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5gzbb"] Jul 07 04:28:09.104736 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.104716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.108313 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.108288 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qxrn7\"" Jul 07 04:28:09.108604 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.108586 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 04:28:09.109620 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.109601 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 04:28:09.130060 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.130035 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5gzbb"] Jul 07 04:28:09.157199 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.157170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.157319 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.157223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26xf\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-kube-api-access-w26xf\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.258503 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.258473 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26xf\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-kube-api-access-w26xf\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.258671 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.258591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.277725 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.277698 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.277913 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.277894 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26xf\" (UniqueName: \"kubernetes.io/projected/d35ede7b-ec5a-4b40-8893-515f1ff34d2f-kube-api-access-w26xf\") pod \"cert-manager-webhook-6486ffd997-5gzbb\" (UID: \"d35ede7b-ec5a-4b40-8893-515f1ff34d2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.422124 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.422032 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:09.561858 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.561832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5gzbb"] Jul 07 04:28:09.564119 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:28:09.564088 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd35ede7b_ec5a_4b40_8893_515f1ff34d2f.slice/crio-951c4bc3ee842f8154b8b54cc626bdf9f9cdee28da96b3ac0890ec90aa0b38cb WatchSource:0}: Error finding container 951c4bc3ee842f8154b8b54cc626bdf9f9cdee28da96b3ac0890ec90aa0b38cb: Status 404 returned error can't find the container with id 951c4bc3ee842f8154b8b54cc626bdf9f9cdee28da96b3ac0890ec90aa0b38cb Jul 07 04:28:09.655812 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:09.655757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" event={"ID":"d35ede7b-ec5a-4b40-8893-515f1ff34d2f","Type":"ContainerStarted","Data":"951c4bc3ee842f8154b8b54cc626bdf9f9cdee28da96b3ac0890ec90aa0b38cb"} Jul 07 04:28:12.671480 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:12.671383 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" event={"ID":"d35ede7b-ec5a-4b40-8893-515f1ff34d2f","Type":"ContainerStarted","Data":"4b38828ea7c99ecb46def6a5a1fc214d5b99e7c2f4089f17c177ae895c5b3f4f"} Jul 07 04:28:12.671480 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:12.671456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:12.694926 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:12.694878 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" podStartSLOduration=0.83819151 podStartE2EDuration="3.694863537s" totalImagesPullingTime="2.856672027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:12.693935791 +0000 UTC m=+265.675869644" watchObservedRunningTime="2026-07-07 04:28:12.694863537 +0000 UTC m=+265.676797389" Jul 07 04:28:15.663030 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.662991 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b"] Jul 07 04:28:15.666189 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.666173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.668654 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.668635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nldbm\"" Jul 07 04:28:15.674826 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.674800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b"] Jul 07 04:28:15.809938 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.809905 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lfl\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-kube-api-access-s4lfl\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.810080 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.809964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.910535 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.910511 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lfl\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-kube-api-access-s4lfl\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.910666 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.910552 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.919566 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.919503 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.919663 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.919610 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lfl\" (UniqueName: \"kubernetes.io/projected/c1baef58-fd51-4929-8807-b71512fd6e85-kube-api-access-s4lfl\") pod \"cert-manager-cainjector-559cdd8c87-dfk5b\" (UID: \"c1baef58-fd51-4929-8807-b71512fd6e85\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:15.975614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:15.975590 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" Jul 07 04:28:16.099196 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:16.099172 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b"] Jul 07 04:28:16.101561 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:28:16.101529 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1baef58_fd51_4929_8807_b71512fd6e85.slice/crio-e9696d93abefc3e7bf7923c044b86703dc54035f89a2193d3b4918963d42182c WatchSource:0}: Error finding container e9696d93abefc3e7bf7923c044b86703dc54035f89a2193d3b4918963d42182c: Status 404 returned error can't find the container with id e9696d93abefc3e7bf7923c044b86703dc54035f89a2193d3b4918963d42182c Jul 07 04:28:16.689787 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:16.689735 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" event={"ID":"c1baef58-fd51-4929-8807-b71512fd6e85","Type":"ContainerStarted","Data":"9a33eb2bffe7fefc45271145e99c071045a7c9fead08b58a519f3e2a6e653418"} Jul 07 04:28:16.689787 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:16.689789 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" event={"ID":"c1baef58-fd51-4929-8807-b71512fd6e85","Type":"ContainerStarted","Data":"e9696d93abefc3e7bf7923c044b86703dc54035f89a2193d3b4918963d42182c"} Jul 07 04:28:16.705913 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:16.705866 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dfk5b" podStartSLOduration=1.705851045 podStartE2EDuration="1.705851045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:16.70522009 +0000 UTC m=+269.687153943" watchObservedRunningTime="2026-07-07 04:28:16.705851045 +0000 UTC m=+269.687784897" Jul 07 04:28:18.677395 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:18.677360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5gzbb" Jul 07 04:28:29.831823 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.831786 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fjwzz"] Jul 07 04:28:29.835032 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.835014 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:29.837362 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.837337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4cc9m\"" Jul 07 04:28:29.845332 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.845309 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fjwzz"] Jul 07 04:28:29.907133 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.907103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwfg\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-kube-api-access-qgwfg\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:29.907238 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:29.907149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-bound-sa-token\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.008336 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.008309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwfg\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-kube-api-access-qgwfg\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.008439 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.008356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-bound-sa-token\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.016242 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.016212 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-bound-sa-token\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.016677 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.016655 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwfg\" (UniqueName: \"kubernetes.io/projected/8a6540da-b865-45a9-9669-6c8c513345df-kube-api-access-qgwfg\") pod \"cert-manager-55b4d954d5-fjwzz\" (UID: \"8a6540da-b865-45a9-9669-6c8c513345df\") " pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.144603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.144534 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fjwzz" Jul 07 04:28:30.270082 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.270045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fjwzz"] Jul 07 04:28:30.272819 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:28:30.272789 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a6540da_b865_45a9_9669_6c8c513345df.slice/crio-3c6e17bbc2b935ed728e03d1599dfe2922422fd0e4275e8304fed361c2560f5f WatchSource:0}: Error finding container 3c6e17bbc2b935ed728e03d1599dfe2922422fd0e4275e8304fed361c2560f5f: Status 404 returned error can't find the container with id 3c6e17bbc2b935ed728e03d1599dfe2922422fd0e4275e8304fed361c2560f5f Jul 07 04:28:30.750318 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.750283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fjwzz" event={"ID":"8a6540da-b865-45a9-9669-6c8c513345df","Type":"ContainerStarted","Data":"f6f67c42a55ead9b9de1e468fa210e7c2f707d91bdf79f1bb0f9cd29fc02089e"} Jul 07 04:28:30.750318 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.750321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fjwzz" event={"ID":"8a6540da-b865-45a9-9669-6c8c513345df","Type":"ContainerStarted","Data":"3c6e17bbc2b935ed728e03d1599dfe2922422fd0e4275e8304fed361c2560f5f"} Jul 07 04:28:30.767986 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:30.767945 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fjwzz" podStartSLOduration=1.767932285 podStartE2EDuration="1.767932285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:30.76661507 +0000 UTC m=+283.748548924" watchObservedRunningTime="2026-07-07 04:28:30.767932285 +0000 UTC m=+283.749866148" Jul 07 04:28:47.227193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.227163 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh"] Jul 07 04:28:47.232868 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.232843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.235262 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.235245 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 04:28:47.235410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.235290 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ftdh2\"" Jul 07 04:28:47.235480 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.235405 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 04:28:47.235694 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.235678 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 04:28:47.235953 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.235932 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 04:28:47.254613 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.254589 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh"] Jul 07 04:28:47.333609 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.333581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.333609 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.333608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.333807 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.333629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.333807 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.333738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.333896 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.333803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgw7\" (UniqueName: \"kubernetes.io/projected/2933a371-7078-4e24-92c2-c252edd93139-kube-api-access-bzgw7\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.432560 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.432530 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:28:47.434115 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.434229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgw7\" (UniqueName: \"kubernetes.io/projected/2933a371-7078-4e24-92c2-c252edd93139-kube-api-access-bzgw7\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.434229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.434229 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.434396 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.434596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.434576 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.436462 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.436440 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 04:28:47.436821 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.436797 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2933a371-7078-4e24-92c2-c252edd93139-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.438020 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.437993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:28:47.441813 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.441762 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 04:28:47.443716 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.443700 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 04:28:47.446927 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.446906 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.447144 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.447118 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2933a371-7078-4e24-92c2-c252edd93139-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.451931 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.451913 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 04:28:47.462684 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.462665 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgw7\" (UniqueName: \"kubernetes.io/projected/2933a371-7078-4e24-92c2-c252edd93139-kube-api-access-bzgw7\") pod \"opendatahub-operator-controller-manager-787b7c66b5-td7dh\" (UID: \"2933a371-7078-4e24-92c2-c252edd93139\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.546056 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.546038 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ftdh2\"" Jul 07 04:28:47.554828 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.554813 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:47.704380 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.704323 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh"] Jul 07 04:28:47.706586 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:28:47.706560 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2933a371_7078_4e24_92c2_c252edd93139.slice/crio-959e711ee306643f0736f302aa9e8acdafe46cf5fcec3f40e49c1319dc4bdb01 WatchSource:0}: Error finding container 959e711ee306643f0736f302aa9e8acdafe46cf5fcec3f40e49c1319dc4bdb01: Status 404 returned error can't find the container with id 959e711ee306643f0736f302aa9e8acdafe46cf5fcec3f40e49c1319dc4bdb01 Jul 07 04:28:47.823384 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:47.823321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" event={"ID":"2933a371-7078-4e24-92c2-c252edd93139","Type":"ContainerStarted","Data":"959e711ee306643f0736f302aa9e8acdafe46cf5fcec3f40e49c1319dc4bdb01"} Jul 07 04:28:50.840327 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:50.840244 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" event={"ID":"2933a371-7078-4e24-92c2-c252edd93139","Type":"ContainerStarted","Data":"5fefca9cd2b965c768018be036aa6a5e261ef156e51b1c1aacdb4db90fd79043"} Jul 07 04:28:51.845978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:51.845940 2563 generic.go:356] "Generic (PLEG): container finished" podID="2933a371-7078-4e24-92c2-c252edd93139" containerID="5fefca9cd2b965c768018be036aa6a5e261ef156e51b1c1aacdb4db90fd79043" exitCode=0 Jul 07 04:28:51.846404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:51.845993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" event={"ID":"2933a371-7078-4e24-92c2-c252edd93139","Type":"ContainerDied","Data":"5fefca9cd2b965c768018be036aa6a5e261ef156e51b1c1aacdb4db90fd79043"} Jul 07 04:28:51.846580 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:51.846560 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:28:52.852863 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:52.852828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" event={"ID":"2933a371-7078-4e24-92c2-c252edd93139","Type":"ContainerStarted","Data":"5a3085aafcc9d5620f29c1399120d3cda11c341a9a6df272f92b9981a99c995b"} Jul 07 04:28:52.853385 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:52.853364 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:28:52.876717 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:28:52.876673 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" podStartSLOduration=1.58500274 podStartE2EDuration="5.876658723s" totalImagesPullingTime="3.210188844s" totalInitContainerRuntime="1.081467139s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:52.874972666 +0000 UTC m=+305.856906518" watchObservedRunningTime="2026-07-07 04:28:52.876658723 +0000 UTC m=+305.858592574" Jul 07 04:29:03.859203 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:03.859178 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-td7dh" Jul 07 04:29:06.797414 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.797374 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm"] Jul 07 04:29:06.805723 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.805700 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.809041 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.809018 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 04:29:06.809142 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.809102 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-txwqx\"" Jul 07 04:29:06.809142 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.809104 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 04:29:06.815891 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.815869 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm"] Jul 07 04:29:06.882513 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.882482 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103c4428-f60a-4302-81a4-62991fc61a55-tmp\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.882513 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.882516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103c4428-f60a-4302-81a4-62991fc61a55-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.882799 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.882550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzhn\" (UniqueName: \"kubernetes.io/projected/103c4428-f60a-4302-81a4-62991fc61a55-kube-api-access-sbzhn\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.983019 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.982969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzhn\" (UniqueName: \"kubernetes.io/projected/103c4428-f60a-4302-81a4-62991fc61a55-kube-api-access-sbzhn\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.983237 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.983063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103c4428-f60a-4302-81a4-62991fc61a55-tmp\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.983237 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.983096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103c4428-f60a-4302-81a4-62991fc61a55-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.985427 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.985398 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103c4428-f60a-4302-81a4-62991fc61a55-tmp\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:06.985904 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:06.985884 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103c4428-f60a-4302-81a4-62991fc61a55-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:07.002419 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:07.002397 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzhn\" (UniqueName: \"kubernetes.io/projected/103c4428-f60a-4302-81a4-62991fc61a55-kube-api-access-sbzhn\") pod \"kube-auth-proxy-7db6fd6878-5k9mm\" (UID: \"103c4428-f60a-4302-81a4-62991fc61a55\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:07.115991 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:07.115921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" Jul 07 04:29:07.243027 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:07.242993 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm"] Jul 07 04:29:07.245989 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:29:07.245963 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c4428_f60a_4302_81a4_62991fc61a55.slice/crio-2d329c920518e32d8c67d7b1deda2a6dd4db649868593f0e30f6f3defd8db222 WatchSource:0}: Error finding container 2d329c920518e32d8c67d7b1deda2a6dd4db649868593f0e30f6f3defd8db222: Status 404 returned error can't find the container with id 2d329c920518e32d8c67d7b1deda2a6dd4db649868593f0e30f6f3defd8db222 Jul 07 04:29:07.921482 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:07.921438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" event={"ID":"103c4428-f60a-4302-81a4-62991fc61a55","Type":"ContainerStarted","Data":"2d329c920518e32d8c67d7b1deda2a6dd4db649868593f0e30f6f3defd8db222"} Jul 07 04:29:10.939268 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:10.939239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" event={"ID":"103c4428-f60a-4302-81a4-62991fc61a55","Type":"ContainerStarted","Data":"bc66100f36340cfc186b5f8d5f92cd516eff1ed85772f17dc23899f86e44c1f1"} Jul 07 04:29:10.957743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:10.957693 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-5k9mm" podStartSLOduration=2.148183164 podStartE2EDuration="4.957677571s" totalImagesPullingTime="2.809494407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:10.955740672 +0000 UTC m=+323.937674524" watchObservedRunningTime="2026-07-07 04:29:10.957677571 +0000 UTC m=+323.939611422" Jul 07 04:29:14.160228 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.160194 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb"] Jul 07 04:29:14.163553 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.163537 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.166069 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.166042 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 04:29:14.166182 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.166108 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8bvdr\"" Jul 07 04:29:14.166182 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.166114 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 04:29:14.166381 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.166367 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 04:29:14.166435 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.166399 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 04:29:14.180801 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.180757 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb"] Jul 07 04:29:14.241764 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241733 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.241920 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.241920 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.241920 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.242053 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.242053 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.241974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79040cef-bf94-40c4-bbca-cd0893cef2cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.242053 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.242025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxn9v\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-kube-api-access-rxn9v\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343343 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343468 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79040cef-bf94-40c4-bbca-cd0893cef2cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxn9v\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-kube-api-access-rxn9v\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.343693 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.343542 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.344265 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.344240 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.345858 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.345831 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.345948 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.345870 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.346547 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.346527 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79040cef-bf94-40c4-bbca-cd0893cef2cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.346740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.346719 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.362850 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.362830 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.362927 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.362903 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxn9v\" (UniqueName: \"kubernetes.io/projected/79040cef-bf94-40c4-bbca-cd0893cef2cd-kube-api-access-rxn9v\") pod \"istiod-openshift-gateway-94bb8fbfd-z8xpb\" (UID: \"79040cef-bf94-40c4-bbca-cd0893cef2cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.472510 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.472430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:14.623270 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.623230 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb"] Jul 07 04:29:14.635371 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:29:14.635330 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79040cef_bf94_40c4_bbca_cd0893cef2cd.slice/crio-992d216e2d2117ed6ec650e19a6419a738bb29eb78c4f3bc6f334e7f240ac515 WatchSource:0}: Error finding container 992d216e2d2117ed6ec650e19a6419a738bb29eb78c4f3bc6f334e7f240ac515: Status 404 returned error can't find the container with id 992d216e2d2117ed6ec650e19a6419a738bb29eb78c4f3bc6f334e7f240ac515 Jul 07 04:29:14.959383 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:14.959338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" event={"ID":"79040cef-bf94-40c4-bbca-cd0893cef2cd","Type":"ContainerStarted","Data":"992d216e2d2117ed6ec650e19a6419a738bb29eb78c4f3bc6f334e7f240ac515"} Jul 07 04:29:16.945429 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:16.945381 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 04:29:16.945719 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:16.945469 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 04:29:17.038156 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.038132 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb"] Jul 07 04:29:17.041522 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.041493 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.043905 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.043833 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 04:29:17.044438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.044411 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j4xxr\"" Jul 07 04:29:17.052137 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.052113 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb"] Jul 07 04:29:17.170914 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.170876 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.171079 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.170952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkk6r\" (UniqueName: \"kubernetes.io/projected/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-kube-api-access-rkk6r\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.271900 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.271867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.272118 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.271955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkk6r\" (UniqueName: \"kubernetes.io/projected/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-kube-api-access-rkk6r\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.274404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.274377 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.310534 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.310505 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkk6r\" (UniqueName: \"kubernetes.io/projected/f9f59874-d250-49e5-9dab-d2c9d93ae8c6-kube-api-access-rkk6r\") pod \"llmisvc-controller-manager-777c6bd7d5-54gzb\" (UID: \"f9f59874-d250-49e5-9dab-d2c9d93ae8c6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.355499 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.355461 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:17.728706 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.728675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb"] Jul 07 04:29:17.731035 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:29:17.731006 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9f59874_d250_49e5_9dab_d2c9d93ae8c6.slice/crio-240bf74dacb68b878a854597feb80fa020e2015f081229fa228df75267e446f1 WatchSource:0}: Error finding container 240bf74dacb68b878a854597feb80fa020e2015f081229fa228df75267e446f1: Status 404 returned error can't find the container with id 240bf74dacb68b878a854597feb80fa020e2015f081229fa228df75267e446f1 Jul 07 04:29:17.980188 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.980108 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" event={"ID":"79040cef-bf94-40c4-bbca-cd0893cef2cd","Type":"ContainerStarted","Data":"3bb89c9d7c260114ba64f92f319e00604a95dc1ad116cc63d787cd2c0b653a11"} Jul 07 04:29:17.980572 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.980222 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:17.981328 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:17.981306 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" event={"ID":"f9f59874-d250-49e5-9dab-d2c9d93ae8c6","Type":"ContainerStarted","Data":"240bf74dacb68b878a854597feb80fa020e2015f081229fa228df75267e446f1"} Jul 07 04:29:18.024529 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:18.024476 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" podStartSLOduration=1.7164186259999998 podStartE2EDuration="4.024461393s" totalImagesPullingTime="2.308042767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:18.023113237 +0000 UTC m=+331.005047074" watchObservedRunningTime="2026-07-07 04:29:18.024461393 +0000 UTC m=+331.006395246" Jul 07 04:29:18.991387 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:18.991345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-z8xpb" Jul 07 04:29:21.000348 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:21.000317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" event={"ID":"f9f59874-d250-49e5-9dab-d2c9d93ae8c6","Type":"ContainerStarted","Data":"eba2219f0dcae497c8a1b4d1c43ecb254accc4a66a2f17ef6e6d3b4fa06e41c4"} Jul 07 04:29:21.000694 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:21.000452 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:29:21.017615 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:21.017569 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" podStartSLOduration=0.970648239 podStartE2EDuration="4.017553753s" totalImagesPullingTime="3.046905514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:21.016411888 +0000 UTC m=+333.998345740" watchObservedRunningTime="2026-07-07 04:29:21.017553753 +0000 UTC m=+333.999487604" Jul 07 04:29:52.006458 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:29:52.006425 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54gzb" Jul 07 04:30:40.240841 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.240812 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk"] Jul 07 04:30:40.245519 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.245502 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:40.248521 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.248504 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n4lbb\"" Jul 07 04:30:40.248588 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.248546 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 04:30:40.249126 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.249110 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 04:30:40.260339 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.260318 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk"] Jul 07 04:30:40.328988 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.328953 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvrp\" (UniqueName: \"kubernetes.io/projected/b2d6b471-8323-4d43-95b8-8f2d3492ca44-kube-api-access-hgvrp\") pod \"authorino-operator-75cc8d5cb7-ksdkk\" (UID: \"b2d6b471-8323-4d43-95b8-8f2d3492ca44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:40.430432 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.430401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvrp\" (UniqueName: \"kubernetes.io/projected/b2d6b471-8323-4d43-95b8-8f2d3492ca44-kube-api-access-hgvrp\") pod \"authorino-operator-75cc8d5cb7-ksdkk\" (UID: \"b2d6b471-8323-4d43-95b8-8f2d3492ca44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:40.441646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.441614 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvrp\" (UniqueName: \"kubernetes.io/projected/b2d6b471-8323-4d43-95b8-8f2d3492ca44-kube-api-access-hgvrp\") pod \"authorino-operator-75cc8d5cb7-ksdkk\" (UID: \"b2d6b471-8323-4d43-95b8-8f2d3492ca44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:40.554868 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.554795 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:40.684551 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:40.684520 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk"] Jul 07 04:30:40.687265 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:30:40.687235 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d6b471_8323_4d43_95b8_8f2d3492ca44.slice/crio-305b95bbd4b36c4a229a1a503798a0763e3936777b4d1b13f569f80dd8823465 WatchSource:0}: Error finding container 305b95bbd4b36c4a229a1a503798a0763e3936777b4d1b13f569f80dd8823465: Status 404 returned error can't find the container with id 305b95bbd4b36c4a229a1a503798a0763e3936777b4d1b13f569f80dd8823465 Jul 07 04:30:41.347610 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:41.347580 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" event={"ID":"b2d6b471-8323-4d43-95b8-8f2d3492ca44","Type":"ContainerStarted","Data":"305b95bbd4b36c4a229a1a503798a0763e3936777b4d1b13f569f80dd8823465"} Jul 07 04:30:43.358318 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:43.358236 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" event={"ID":"b2d6b471-8323-4d43-95b8-8f2d3492ca44","Type":"ContainerStarted","Data":"a78019a0c19c96d7d799fae7886594e8bdcf8a0b40c09984e5a2a9c2d24d7b3e"} Jul 07 04:30:43.358655 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:43.358344 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:30:43.384021 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:43.383965 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" podStartSLOduration=0.963916553 podStartE2EDuration="3.383832593s" totalImagesPullingTime="2.41991604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:30:43.37653513 +0000 UTC m=+416.358468994" watchObservedRunningTime="2026-07-07 04:30:43.383832593 +0000 UTC m=+416.365766446" Jul 07 04:30:50.834170 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:50.834132 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm"] Jul 07 04:30:50.837645 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:50.837626 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:50.840103 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:50.840079 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9r79q\"" Jul 07 04:30:50.847165 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:50.847143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm"] Jul 07 04:30:51.018964 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.018933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rxj\" (UniqueName: \"kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj\") pod \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" (UID: \"554ea3bb-d10e-457d-9f53-40f30833e9db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:51.120292 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.120211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rxj\" (UniqueName: \"kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj\") pod \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" (UID: \"554ea3bb-d10e-457d-9f53-40f30833e9db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:51.128268 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.128242 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rxj\" (UniqueName: \"kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj\") pod \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" (UID: \"554ea3bb-d10e-457d-9f53-40f30833e9db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:51.149154 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.149132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:51.275097 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.275075 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm"] Jul 07 04:30:51.277818 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:30:51.277790 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554ea3bb_d10e_457d_9f53_40f30833e9db.slice/crio-ff5bb821f58150adb5ba08c62be759b83080d4158470bcb775885dc8115eb06a WatchSource:0}: Error finding container ff5bb821f58150adb5ba08c62be759b83080d4158470bcb775885dc8115eb06a: Status 404 returned error can't find the container with id ff5bb821f58150adb5ba08c62be759b83080d4158470bcb775885dc8115eb06a Jul 07 04:30:51.400569 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:51.400498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" event={"ID":"554ea3bb-d10e-457d-9f53-40f30833e9db","Type":"ContainerStarted","Data":"ff5bb821f58150adb5ba08c62be759b83080d4158470bcb775885dc8115eb06a"} Jul 07 04:30:53.411907 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:53.411827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" event={"ID":"554ea3bb-d10e-457d-9f53-40f30833e9db","Type":"ContainerStarted","Data":"3322b7a7b846f8dce5a01987711f7b1ac181e310878238a4bc1154677ca741b7"} Jul 07 04:30:53.412266 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:53.411976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:30:53.432321 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:53.432034 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" podStartSLOduration=1.623503924 podStartE2EDuration="3.432015766s" totalImagesPullingTime="1.808511842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:30:53.431016542 +0000 UTC m=+426.412950407" watchObservedRunningTime="2026-07-07 04:30:53.432015766 +0000 UTC m=+426.413949621" Jul 07 04:30:54.364574 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:30:54.364545 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ksdkk" Jul 07 04:31:04.419465 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:04.419435 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:05.925811 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.925763 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq"] Jul 07 04:31:05.929190 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.929173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:05.929905 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.929883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkwj\" (UniqueName: \"kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:05.930003 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.929933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:05.931872 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.931850 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m5r5j\"" Jul 07 04:31:05.943985 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:05.943960 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq"] Jul 07 04:31:06.030684 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.030661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.030805 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.030710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkwj\" (UniqueName: \"kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.031054 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.031036 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.041279 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.041253 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkwj\" (UniqueName: \"kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj\") pod \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.239047 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.239022 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.284442 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.284413 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq"] Jul 07 04:31:06.299323 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.299259 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq"] Jul 07 04:31:06.350694 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.350660 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm"] Jul 07 04:31:06.351191 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.350921 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" containerName="manager" containerID="cri-o://3322b7a7b846f8dce5a01987711f7b1ac181e310878238a4bc1154677ca741b7" gracePeriod=2 Jul 07 04:31:06.359976 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.359951 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm"] Jul 07 04:31:06.374217 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.374194 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7"] Jul 07 04:31:06.374526 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.374513 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" containerName="manager" Jul 07 04:31:06.374573 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.374528 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" containerName="manager" Jul 07 04:31:06.374607 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.374601 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" containerName="manager" Jul 07 04:31:06.377515 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.377493 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:06.385680 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.385653 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7"] Jul 07 04:31:06.407915 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:06.407878 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:06.432792 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.432751 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxptl\" (UniqueName: \"kubernetes.io/projected/f4413624-b0c1-4510-8c73-17aa22855626-kube-api-access-wxptl\") pod \"limitador-operator-controller-manager-859d7b78bd-vblh7\" (UID: \"f4413624-b0c1-4510-8c73-17aa22855626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:06.470993 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.470953 2563 generic.go:356] "Generic (PLEG): container finished" podID="554ea3bb-d10e-457d-9f53-40f30833e9db" containerID="3322b7a7b846f8dce5a01987711f7b1ac181e310878238a4bc1154677ca741b7" exitCode=0 Jul 07 04:31:06.518077 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:06.518041 2563 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 07 04:31:06.518077 ip-10-0-142-140 kubenswrapper[2563]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-8j7sq_kuadrant-system_be862c68-f770-46b0-87e0-6dcaeb12a6e9_0(852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-8j7sq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8" Netns:"/var/run/netns/4c18ba15-c973-41b4-831c-ca0181620733" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-8j7sq;K8S_POD_INFRA_CONTAINER_ID=852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8;K8S_POD_UID=be862c68-f770-46b0-87e0-6dcaeb12a6e9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq/be862c68-f770-46b0-87e0-6dcaeb12a6e9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq 852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq 852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:20 [10.132.0.32/23] Jul 07 04:31:06.518077 ip-10-0-142-140 kubenswrapper[2563]: ' Jul 07 04:31:06.518077 ip-10-0-142-140 kubenswrapper[2563]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 07 04:31:06.518077 ip-10-0-142-140 kubenswrapper[2563]: > Jul 07 04:31:06.518302 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:06.518133 2563 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 07 04:31:06.518302 ip-10-0-142-140 kubenswrapper[2563]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-8j7sq_kuadrant-system_be862c68-f770-46b0-87e0-6dcaeb12a6e9_0(852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-8j7sq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8" Netns:"/var/run/netns/4c18ba15-c973-41b4-831c-ca0181620733" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-8j7sq;K8S_POD_INFRA_CONTAINER_ID=852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8;K8S_POD_UID=be862c68-f770-46b0-87e0-6dcaeb12a6e9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq/be862c68-f770-46b0-87e0-6dcaeb12a6e9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq 852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8 network default NAD default NAD key default] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq 852a3eccafd2b4f01501d8e28aa45467a4ea5a067f767c8f23e921cd5196a8d8 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:20 [10.132.0.32/23] Jul 07 04:31:06.518302 ip-10-0-142-140 kubenswrapper[2563]: ' Jul 07 04:31:06.518302 ip-10-0-142-140 kubenswrapper[2563]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 07 04:31:06.518302 ip-10-0-142-140 kubenswrapper[2563]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:06.533423 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.533400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxptl\" (UniqueName: \"kubernetes.io/projected/f4413624-b0c1-4510-8c73-17aa22855626-kube-api-access-wxptl\") pod \"limitador-operator-controller-manager-859d7b78bd-vblh7\" (UID: \"f4413624-b0c1-4510-8c73-17aa22855626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:06.545529 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.545507 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxptl\" (UniqueName: \"kubernetes.io/projected/f4413624-b0c1-4510-8c73-17aa22855626-kube-api-access-wxptl\") pod \"limitador-operator-controller-manager-859d7b78bd-vblh7\" (UID: \"f4413624-b0c1-4510-8c73-17aa22855626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:06.577815 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.577795 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:06.579869 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:06.579846 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:06.634172 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.634146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2rxj\" (UniqueName: \"kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj\") pod \"554ea3bb-d10e-457d-9f53-40f30833e9db\" (UID: \"554ea3bb-d10e-457d-9f53-40f30833e9db\") " Jul 07 04:31:06.636119 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.636098 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj" (OuterVolumeSpecName: "kube-api-access-h2rxj") pod "554ea3bb-d10e-457d-9f53-40f30833e9db" (UID: "554ea3bb-d10e-457d-9f53-40f30833e9db"). InnerVolumeSpecName "kube-api-access-h2rxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:31:06.722435 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.722411 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:06.735472 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.735453 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rxj\" (UniqueName: \"kubernetes.io/projected/554ea3bb-d10e-457d-9f53-40f30833e9db-kube-api-access-h2rxj\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:31:06.852921 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:06.852705 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7"] Jul 07 04:31:06.856219 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:31:06.856180 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4413624_b0c1_4510_8c73_17aa22855626.slice/crio-1bbffde3cf3959c2b2c08de1b8b776ee05faa52254869f127a685b2fea4e5b42 WatchSource:0}: Error finding container 1bbffde3cf3959c2b2c08de1b8b776ee05faa52254869f127a685b2fea4e5b42: Status 404 returned error can't find the container with id 1bbffde3cf3959c2b2c08de1b8b776ee05faa52254869f127a685b2fea4e5b42 Jul 07 04:31:07.477431 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.477342 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:07.477431 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.477369 2563 scope.go:117] "RemoveContainer" containerID="3322b7a7b846f8dce5a01987711f7b1ac181e310878238a4bc1154677ca741b7" Jul 07 04:31:07.479145 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.479112 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" event={"ID":"f4413624-b0c1-4510-8c73-17aa22855626","Type":"ContainerStarted","Data":"a17397c9520929c40114273d446d3b8fcc027b38f8b62ee69aeba534ab0a97de"} Jul 07 04:31:07.479145 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.479132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:07.479326 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.479183 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" event={"ID":"f4413624-b0c1-4510-8c73-17aa22855626","Type":"ContainerStarted","Data":"1bbffde3cf3959c2b2c08de1b8b776ee05faa52254869f127a685b2fea4e5b42"} Jul 07 04:31:07.479326 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.479228 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:31:07.479830 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.479805 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:07.481970 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.481936 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:07.483834 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.483804 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="be862c68-f770-46b0-87e0-6dcaeb12a6e9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:07.487519 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.487505 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:07.504243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.504199 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" podStartSLOduration=1.504186234 podStartE2EDuration="1.504186234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:31:07.503250936 +0000 UTC m=+440.485184779" watchObservedRunningTime="2026-07-07 04:31:07.504186234 +0000 UTC m=+440.486120115" Jul 07 04:31:07.505112 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.505087 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="be862c68-f770-46b0-87e0-6dcaeb12a6e9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:07.506966 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.506943 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:07.526642 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.526618 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="be862c68-f770-46b0-87e0-6dcaeb12a6e9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:07.527615 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.527596 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" path="/var/lib/kubelet/pods/554ea3bb-d10e-457d-9f53-40f30833e9db/volumes" Jul 07 04:31:07.529030 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:07.529008 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kbwdm\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="554ea3bb-d10e-457d-9f53-40f30833e9db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kbwdm" Jul 07 04:31:07.541502 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.541484 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume\") pod \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " Jul 07 04:31:07.541596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.541545 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnkwj\" (UniqueName: \"kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj\") pod \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\" (UID: \"be862c68-f770-46b0-87e0-6dcaeb12a6e9\") " Jul 07 04:31:07.541744 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.541722 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "be862c68-f770-46b0-87e0-6dcaeb12a6e9" (UID: "be862c68-f770-46b0-87e0-6dcaeb12a6e9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:31:07.543547 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.543519 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj" (OuterVolumeSpecName: "kube-api-access-hnkwj") pod "be862c68-f770-46b0-87e0-6dcaeb12a6e9" (UID: "be862c68-f770-46b0-87e0-6dcaeb12a6e9"). InnerVolumeSpecName "kube-api-access-hnkwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:31:07.642664 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.642634 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be862c68-f770-46b0-87e0-6dcaeb12a6e9-extensions-socket-volume\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:31:07.642664 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:07.642660 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkwj\" (UniqueName: \"kubernetes.io/projected/be862c68-f770-46b0-87e0-6dcaeb12a6e9-kube-api-access-hnkwj\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:31:08.485036 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:08.485008 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:08.492751 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:31:08.492724 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8j7sq\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="be862c68-f770-46b0-87e0-6dcaeb12a6e9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8j7sq" Jul 07 04:31:09.527081 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:09.527051 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be862c68-f770-46b0-87e0-6dcaeb12a6e9" path="/var/lib/kubelet/pods/be862c68-f770-46b0-87e0-6dcaeb12a6e9/volumes" Jul 07 04:31:18.487534 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:31:18.487455 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vblh7" Jul 07 04:32:33.024963 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.024926 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:33.028398 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.028373 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.031854 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.031832 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 04:32:33.031985 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.031832 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cpsxj\"" Jul 07 04:32:33.032320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.032301 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 04:32:33.040016 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.039993 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:33.098736 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.098699 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfr7\" (UniqueName: \"kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.098924 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.098827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.199206 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.199179 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:32:33.199531 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.199504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.199601 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.199557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfr7\" (UniqueName: \"kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.199666 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:32:33.199649 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 07 04:32:33.199722 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:32:33.199713 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert podName:07962c95-5a5e-4ea6-b058-d88ca48618f6 nodeName:}" failed. No retries permitted until 2026-07-07 04:32:33.699699533 +0000 UTC m=+526.681633363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert") pod "maas-controller-6d57bc4d78-t6g44" (UID: "07962c95-5a5e-4ea6-b058-d88ca48618f6") : secret "maas-controller-webhook-cert" not found Jul 07 04:32:33.202490 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.202472 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.211786 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.211750 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfr7\" (UniqueName: \"kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.213614 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.213592 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:32:33.300209 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.300135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj8g\" (UniqueName: \"kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.300325 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.300238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.401116 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.401086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.401286 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.401140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj8g\" (UniqueName: \"kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.403522 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.403498 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.410070 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.410051 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj8g\" (UniqueName: \"kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g\") pod \"maas-controller-65fd5f6bc4-xcwqb\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.512859 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.512832 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:33.639830 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.639802 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:32:33.642718 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:32:33.642693 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12c512c_b01d_4573_961f_42eb5858bccc.slice/crio-4b14b2a26012cc0e35c9bb8e67b8f4ce214295500f0408be8ea71439ec549b71 WatchSource:0}: Error finding container 4b14b2a26012cc0e35c9bb8e67b8f4ce214295500f0408be8ea71439ec549b71: Status 404 returned error can't find the container with id 4b14b2a26012cc0e35c9bb8e67b8f4ce214295500f0408be8ea71439ec549b71 Jul 07 04:32:33.704484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.704458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.706646 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.706619 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") pod \"maas-controller-6d57bc4d78-t6g44\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:33.862950 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.862879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" event={"ID":"c12c512c-b01d-4573-961f-42eb5858bccc","Type":"ContainerStarted","Data":"4b14b2a26012cc0e35c9bb8e67b8f4ce214295500f0408be8ea71439ec549b71"} Jul 07 04:32:33.939720 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:33.939694 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:34.071894 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:34.071862 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:34.074722 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:32:34.074697 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07962c95_5a5e_4ea6_b058_d88ca48618f6.slice/crio-bd3e5afcab3f5ecd00c17d5dd1a655c4b9ae740064f17ab437d84ac24c261e88 WatchSource:0}: Error finding container bd3e5afcab3f5ecd00c17d5dd1a655c4b9ae740064f17ab437d84ac24c261e88: Status 404 returned error can't find the container with id bd3e5afcab3f5ecd00c17d5dd1a655c4b9ae740064f17ab437d84ac24c261e88 Jul 07 04:32:34.872215 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:34.872171 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" event={"ID":"07962c95-5a5e-4ea6-b058-d88ca48618f6","Type":"ContainerStarted","Data":"bd3e5afcab3f5ecd00c17d5dd1a655c4b9ae740064f17ab437d84ac24c261e88"} Jul 07 04:32:36.882782 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.882745 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" event={"ID":"07962c95-5a5e-4ea6-b058-d88ca48618f6","Type":"ContainerStarted","Data":"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828"} Jul 07 04:32:36.883222 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.882844 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:36.884270 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.884246 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" event={"ID":"c12c512c-b01d-4573-961f-42eb5858bccc","Type":"ContainerStarted","Data":"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f"} Jul 07 04:32:36.884389 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.884369 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:36.904466 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.904424 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" podStartSLOduration=3.0710042 podStartE2EDuration="4.904408965s" totalImagesPullingTime="1.833404765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:32:36.903415332 +0000 UTC m=+529.885349184" watchObservedRunningTime="2026-07-07 04:32:36.904408965 +0000 UTC m=+529.886342816" Jul 07 04:32:36.922585 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:36.922546 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" podStartSLOduration=1.668294942 podStartE2EDuration="3.922515261s" totalImagesPullingTime="2.254220319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:32:36.921420482 +0000 UTC m=+529.903354336" watchObservedRunningTime="2026-07-07 04:32:36.922515261 +0000 UTC m=+529.904449112" Jul 07 04:32:47.895115 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:47.895087 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:47.895558 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:47.895154 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:32:47.990097 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:47.990065 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:47.990263 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:47.990229 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" podUID="07962c95-5a5e-4ea6-b058-d88ca48618f6" containerName="manager" containerID="cri-o://43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828" gracePeriod=30 Jul 07 04:32:48.239164 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.239132 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:48.318027 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.317995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfr7\" (UniqueName: \"kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7\") pod \"07962c95-5a5e-4ea6-b058-d88ca48618f6\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " Jul 07 04:32:48.318181 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.318045 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") pod \"07962c95-5a5e-4ea6-b058-d88ca48618f6\" (UID: \"07962c95-5a5e-4ea6-b058-d88ca48618f6\") " Jul 07 04:32:48.320230 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.320200 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7" (OuterVolumeSpecName: "kube-api-access-6lfr7") pod "07962c95-5a5e-4ea6-b058-d88ca48618f6" (UID: "07962c95-5a5e-4ea6-b058-d88ca48618f6"). InnerVolumeSpecName "kube-api-access-6lfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:32:48.320300 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.320240 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "07962c95-5a5e-4ea6-b058-d88ca48618f6" (UID: "07962c95-5a5e-4ea6-b058-d88ca48618f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:32:48.419032 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.418944 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfr7\" (UniqueName: \"kubernetes.io/projected/07962c95-5a5e-4ea6-b058-d88ca48618f6-kube-api-access-6lfr7\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:32:48.419032 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.418976 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07962c95-5a5e-4ea6-b058-d88ca48618f6-webhook-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:32:48.938533 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.938495 2563 generic.go:356] "Generic (PLEG): container finished" podID="07962c95-5a5e-4ea6-b058-d88ca48618f6" containerID="43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828" exitCode=0 Jul 07 04:32:48.938533 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.938536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" Jul 07 04:32:48.939066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.938540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" event={"ID":"07962c95-5a5e-4ea6-b058-d88ca48618f6","Type":"ContainerDied","Data":"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828"} Jul 07 04:32:48.939066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.938654 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d57bc4d78-t6g44" event={"ID":"07962c95-5a5e-4ea6-b058-d88ca48618f6","Type":"ContainerDied","Data":"bd3e5afcab3f5ecd00c17d5dd1a655c4b9ae740064f17ab437d84ac24c261e88"} Jul 07 04:32:48.939066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.938692 2563 scope.go:117] "RemoveContainer" containerID="43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828" Jul 07 04:32:48.947705 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.947686 2563 scope.go:117] "RemoveContainer" containerID="43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828" Jul 07 04:32:48.948001 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:32:48.947983 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828\": container with ID starting with 43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828 not found: ID does not exist" containerID="43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828" Jul 07 04:32:48.948043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.948010 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828"} err="failed to get container status \"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828\": rpc error: code = NotFound desc = could not find container \"43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828\": container with ID starting with 43947cfdcdbfd9224a7591cf7a82fcccfe0064b01836b13ecb0cf245a5ba4828 not found: ID does not exist" Jul 07 04:32:48.960729 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.960690 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:48.965040 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:48.965016 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d57bc4d78-t6g44"] Jul 07 04:32:49.527806 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:32:49.527751 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07962c95-5a5e-4ea6-b058-d88ca48618f6" path="/var/lib/kubelet/pods/07962c95-5a5e-4ea6-b058-d88ca48618f6/volumes" Jul 07 04:33:08.267425 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.267389 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-69668d87b5-7sgxk"] Jul 07 04:33:08.267885 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.267866 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07962c95-5a5e-4ea6-b058-d88ca48618f6" containerName="manager" Jul 07 04:33:08.267938 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.267889 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07962c95-5a5e-4ea6-b058-d88ca48618f6" containerName="manager" Jul 07 04:33:08.267997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.267986 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07962c95-5a5e-4ea6-b058-d88ca48618f6" containerName="manager" Jul 07 04:33:08.272347 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.272324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.274531 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.274508 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 04:33:08.274633 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.274530 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-fjrxt\"" Jul 07 04:33:08.282429 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.282402 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69668d87b5-7sgxk"] Jul 07 04:33:08.372944 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.372913 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.373079 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.372952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5zk\" (UniqueName: \"kubernetes.io/projected/8651ce06-ffbc-4759-a217-49972bf1ffe2-kube-api-access-qj5zk\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.373079 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.373040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8651ce06-ffbc-4759-a217-49972bf1ffe2-openshift-service-ca\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.474495 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.474448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8651ce06-ffbc-4759-a217-49972bf1ffe2-openshift-service-ca\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.474698 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.474530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.474698 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.474570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5zk\" (UniqueName: \"kubernetes.io/projected/8651ce06-ffbc-4759-a217-49972bf1ffe2-kube-api-access-qj5zk\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.474698 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:33:08.474674 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 07 04:33:08.474901 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:33:08.474740 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls podName:8651ce06-ffbc-4759-a217-49972bf1ffe2 nodeName:}" failed. No retries permitted until 2026-07-07 04:33:08.974722114 +0000 UTC m=+561.956655949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls") pod "maas-api-69668d87b5-7sgxk" (UID: "8651ce06-ffbc-4759-a217-49972bf1ffe2") : secret "maas-api-serving-cert" not found Jul 07 04:33:08.475187 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.475167 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8651ce06-ffbc-4759-a217-49972bf1ffe2-openshift-service-ca\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.483880 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.483855 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5zk\" (UniqueName: \"kubernetes.io/projected/8651ce06-ffbc-4759-a217-49972bf1ffe2-kube-api-access-qj5zk\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.978508 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.978469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:08.980850 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:08.980830 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8651ce06-ffbc-4759-a217-49972bf1ffe2-maas-api-tls\") pod \"maas-api-69668d87b5-7sgxk\" (UID: \"8651ce06-ffbc-4759-a217-49972bf1ffe2\") " pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:09.183300 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:09.183265 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:09.319735 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:09.319681 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69668d87b5-7sgxk"] Jul 07 04:33:09.322208 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:33:09.322178 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8651ce06_ffbc_4759_a217_49972bf1ffe2.slice/crio-be267d81e7c2edc05aabe97e820cff733f1254def870ceec41740740a0d5ff2e WatchSource:0}: Error finding container be267d81e7c2edc05aabe97e820cff733f1254def870ceec41740740a0d5ff2e: Status 404 returned error can't find the container with id be267d81e7c2edc05aabe97e820cff733f1254def870ceec41740740a0d5ff2e Jul 07 04:33:10.047929 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:10.047884 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69668d87b5-7sgxk" event={"ID":"8651ce06-ffbc-4759-a217-49972bf1ffe2","Type":"ContainerStarted","Data":"be267d81e7c2edc05aabe97e820cff733f1254def870ceec41740740a0d5ff2e"} Jul 07 04:33:11.053780 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:11.053733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69668d87b5-7sgxk" event={"ID":"8651ce06-ffbc-4759-a217-49972bf1ffe2","Type":"ContainerStarted","Data":"fae4dd29b6611a84348433654b3748bc2b40e5fab40f306365a250d1d294a252"} Jul 07 04:33:11.054142 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:11.053871 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:11.076292 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:11.076204 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-69668d87b5-7sgxk" podStartSLOduration=1.584563181 podStartE2EDuration="3.07618548s" totalImagesPullingTime="1.491622299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:33:11.075132579 +0000 UTC m=+564.057066432" watchObservedRunningTime="2026-07-07 04:33:11.07618548 +0000 UTC m=+564.058119332" Jul 07 04:33:17.064412 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:17.064384 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-69668d87b5-7sgxk" Jul 07 04:33:32.897504 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.897472 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2"] Jul 07 04:33:32.901255 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.901232 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.904491 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.904471 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 04:33:32.904608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.904508 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 04:33:32.904608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.904564 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 04:33:32.904608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.904471 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xnb2g\"" Jul 07 04:33:32.910964 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.910942 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2"] Jul 07 04:33:32.972835 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.972804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.972997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.972848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.972997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.972910 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb0837e-669d-4e4f-8827-65d671f97e2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.972997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.972980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67ps\" (UniqueName: \"kubernetes.io/projected/cbb0837e-669d-4e4f-8827-65d671f97e2c-kube-api-access-n67ps\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.973140 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.973042 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:32.973140 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:32.973065 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074251 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67ps\" (UniqueName: \"kubernetes.io/projected/cbb0837e-669d-4e4f-8827-65d671f97e2c-kube-api-access-n67ps\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074328 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074343 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074360 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb0837e-669d-4e4f-8827-65d671f97e2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074795 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074746 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074863 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074764 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.074907 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.074862 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.076573 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.076553 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb0837e-669d-4e4f-8827-65d671f97e2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.076685 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.076669 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb0837e-669d-4e4f-8827-65d671f97e2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.081873 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.081850 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67ps\" (UniqueName: \"kubernetes.io/projected/cbb0837e-669d-4e4f-8827-65d671f97e2c-kube-api-access-n67ps\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2\" (UID: \"cbb0837e-669d-4e4f-8827-65d671f97e2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.213099 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.213021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:33.349039 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:33.349014 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2"] Jul 07 04:33:33.351487 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:33:33.351461 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb0837e_669d_4e4f_8827_65d671f97e2c.slice/crio-a679039d71e38bb8090dad9a178001625de843fd287c68fd6844ea2c2513b58d WatchSource:0}: Error finding container a679039d71e38bb8090dad9a178001625de843fd287c68fd6844ea2c2513b58d: Status 404 returned error can't find the container with id a679039d71e38bb8090dad9a178001625de843fd287c68fd6844ea2c2513b58d Jul 07 04:33:34.160450 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:34.160416 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" event={"ID":"cbb0837e-669d-4e4f-8827-65d671f97e2c","Type":"ContainerStarted","Data":"a679039d71e38bb8090dad9a178001625de843fd287c68fd6844ea2c2513b58d"} Jul 07 04:33:35.166157 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:35.166123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" event={"ID":"cbb0837e-669d-4e4f-8827-65d671f97e2c","Type":"ContainerStarted","Data":"2a5279b65e10de3663ae8f2fb0b48b5e6ea4e67540c9d8dc996cf68733e45b22"} Jul 07 04:33:35.185126 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:35.185070 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" podStartSLOduration=2.069743137 podStartE2EDuration="3.18505034s" totalImagesPullingTime="1.115307203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:33:35.184705506 +0000 UTC m=+588.166639370" watchObservedRunningTime="2026-07-07 04:33:35.18505034 +0000 UTC m=+588.166984193" Jul 07 04:33:43.214140 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:43.214105 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:43.214574 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:43.214161 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:43.226647 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:43.226626 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:44.217671 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:44.217642 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2" Jul 07 04:33:47.463758 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:47.463724 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:33:47.466002 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:33:47.465976 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:34:33.188444 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.188404 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9"] Jul 07 04:34:33.191997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.191974 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.194674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.194654 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 04:34:33.201939 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.201919 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9"] Jul 07 04:34:33.340427 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.340427 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.340632 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340460 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.340632 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.340632 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.340632 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.340626 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgrv\" (UniqueName: \"kubernetes.io/projected/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-kube-api-access-4qgrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441452 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441452 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441452 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441706 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441706 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441519 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441706 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgrv\" (UniqueName: \"kubernetes.io/projected/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-kube-api-access-4qgrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.441953 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441930 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.442030 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441973 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.442030 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.441981 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.443729 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.443705 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.443932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.443914 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.450551 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.450526 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgrv\" (UniqueName: \"kubernetes.io/projected/e56a4cc3-1d34-4256-a86b-c49f9f6f07bc-kube-api-access-4qgrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9\" (UID: \"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.502127 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.502107 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:33.638197 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.638171 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9"] Jul 07 04:34:33.640500 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:34:33.640473 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56a4cc3_1d34_4256_a86b_c49f9f6f07bc.slice/crio-c9d739db33da6e617c785e2da2d5d2c95abbc997f3af489aca6f105f9b279759 WatchSource:0}: Error finding container c9d739db33da6e617c785e2da2d5d2c95abbc997f3af489aca6f105f9b279759: Status 404 returned error can't find the container with id c9d739db33da6e617c785e2da2d5d2c95abbc997f3af489aca6f105f9b279759 Jul 07 04:34:33.641901 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:33.641885 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:34:34.440577 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:34.440542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" event={"ID":"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc","Type":"ContainerStarted","Data":"68aa4c2fc3b9aabcb4bc89071290278935954df5176003fd94446eb03ccfa567"} Jul 07 04:34:34.440944 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:34.440593 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" event={"ID":"e56a4cc3-1d34-4256-a86b-c49f9f6f07bc","Type":"ContainerStarted","Data":"c9d739db33da6e617c785e2da2d5d2c95abbc997f3af489aca6f105f9b279759"} Jul 07 04:34:34.460611 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:34.460566 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" podStartSLOduration=1.2813002359999999 podStartE2EDuration="1.460549118s" totalImagesPullingTime="179.248882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:34:34.458430253 +0000 UTC m=+647.440364102" watchObservedRunningTime="2026-07-07 04:34:34.460549118 +0000 UTC m=+647.442482969" Jul 07 04:34:41.910071 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:41.910041 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs"] Jul 07 04:34:41.913468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:41.913450 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:41.917856 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:41.917817 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 04:34:41.930832 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:41.930810 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs"] Jul 07 04:34:42.009257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009227 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.009443 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.009443 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.009443 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.009562 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009477 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7972eb63-782d-4ae5-86b5-0a5e6564c695-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.009562 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.009508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5bd\" (UniqueName: \"kubernetes.io/projected/7972eb63-782d-4ae5-86b5-0a5e6564c695-kube-api-access-7j5bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110293 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110293 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110502 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7972eb63-782d-4ae5-86b5-0a5e6564c695-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110502 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5bd\" (UniqueName: \"kubernetes.io/projected/7972eb63-782d-4ae5-86b5-0a5e6564c695-kube-api-access-7j5bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110857 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110828 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110954 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110916 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.110954 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.110939 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.112643 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.112614 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7972eb63-782d-4ae5-86b5-0a5e6564c695-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.112864 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.112848 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7972eb63-782d-4ae5-86b5-0a5e6564c695-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.119892 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.119866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5bd\" (UniqueName: \"kubernetes.io/projected/7972eb63-782d-4ae5-86b5-0a5e6564c695-kube-api-access-7j5bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gcsxs\" (UID: \"7972eb63-782d-4ae5-86b5-0a5e6564c695\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.224562 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.224471 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:42.354033 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.354004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs"] Jul 07 04:34:42.355567 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:34:42.355542 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7972eb63_782d_4ae5_86b5_0a5e6564c695.slice/crio-4f63f29b3bc777815f5a087df03973f56865d4b24341e3c751ab4bbd018e203b WatchSource:0}: Error finding container 4f63f29b3bc777815f5a087df03973f56865d4b24341e3c751ab4bbd018e203b: Status 404 returned error can't find the container with id 4f63f29b3bc777815f5a087df03973f56865d4b24341e3c751ab4bbd018e203b Jul 07 04:34:42.481121 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:42.481055 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" event={"ID":"7972eb63-782d-4ae5-86b5-0a5e6564c695","Type":"ContainerStarted","Data":"4f63f29b3bc777815f5a087df03973f56865d4b24341e3c751ab4bbd018e203b"} Jul 07 04:34:43.487542 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:43.487503 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" event={"ID":"7972eb63-782d-4ae5-86b5-0a5e6564c695","Type":"ContainerStarted","Data":"ea6954a5a6dad668997bfb08ba1756b7c36408843e2137c6112f022214a6d571"} Jul 07 04:34:43.503119 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:43.503098 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:43.503362 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:43.503337 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:43.504906 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:43.504858 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" podStartSLOduration=2.297465942 podStartE2EDuration="2.504839961s" totalImagesPullingTime="207.374019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:34:43.504050194 +0000 UTC m=+656.485984059" watchObservedRunningTime="2026-07-07 04:34:43.504839961 +0000 UTC m=+656.486773816" Jul 07 04:34:43.516075 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:43.516056 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:44.503527 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:44.503497 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9" Jul 07 04:34:52.225458 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:52.225421 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:52.225959 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:52.225485 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:52.238198 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:52.238176 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:34:52.539673 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:34:52.539648 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gcsxs" Jul 07 04:36:51.803484 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:51.803451 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:36:51.803989 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:51.803649 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" podUID="c12c512c-b01d-4573-961f-42eb5858bccc" containerName="manager" containerID="cri-o://47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f" gracePeriod=30 Jul 07 04:36:52.045047 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.045025 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:36:52.097268 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.097180 2563 generic.go:356] "Generic (PLEG): container finished" podID="c12c512c-b01d-4573-961f-42eb5858bccc" containerID="47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f" exitCode=0 Jul 07 04:36:52.097268 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.097237 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" Jul 07 04:36:52.097448 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.097273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" event={"ID":"c12c512c-b01d-4573-961f-42eb5858bccc","Type":"ContainerDied","Data":"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f"} Jul 07 04:36:52.097448 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.097338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-xcwqb" event={"ID":"c12c512c-b01d-4573-961f-42eb5858bccc","Type":"ContainerDied","Data":"4b14b2a26012cc0e35c9bb8e67b8f4ce214295500f0408be8ea71439ec549b71"} Jul 07 04:36:52.097448 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.097365 2563 scope.go:117] "RemoveContainer" containerID="47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f" Jul 07 04:36:52.106224 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.106196 2563 scope.go:117] "RemoveContainer" containerID="47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f" Jul 07 04:36:52.106486 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:36:52.106467 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f\": container with ID starting with 47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f not found: ID does not exist" containerID="47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f" Jul 07 04:36:52.106550 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.106495 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f"} err="failed to get container status \"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f\": rpc error: code = NotFound desc = could not find container \"47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f\": container with ID starting with 47f79efe2069bc1859d9d6c278d7782a2a120bcea629b59ef9a01d09e929453f not found: ID does not exist" Jul 07 04:36:52.157353 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.157328 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzj8g\" (UniqueName: \"kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g\") pod \"c12c512c-b01d-4573-961f-42eb5858bccc\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " Jul 07 04:36:52.157477 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.157369 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert\") pod \"c12c512c-b01d-4573-961f-42eb5858bccc\" (UID: \"c12c512c-b01d-4573-961f-42eb5858bccc\") " Jul 07 04:36:52.159364 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.159338 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c12c512c-b01d-4573-961f-42eb5858bccc" (UID: "c12c512c-b01d-4573-961f-42eb5858bccc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:36:52.159460 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.159385 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g" (OuterVolumeSpecName: "kube-api-access-mzj8g") pod "c12c512c-b01d-4573-961f-42eb5858bccc" (UID: "c12c512c-b01d-4573-961f-42eb5858bccc"). InnerVolumeSpecName "kube-api-access-mzj8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:36:52.258596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.258569 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzj8g\" (UniqueName: \"kubernetes.io/projected/c12c512c-b01d-4573-961f-42eb5858bccc-kube-api-access-mzj8g\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:36:52.258596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.258594 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c12c512c-b01d-4573-961f-42eb5858bccc-webhook-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:36:52.421043 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.421012 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:36:52.424954 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:52.424921 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-xcwqb"] Jul 07 04:36:53.186603 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.186570 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-ll7cz"] Jul 07 04:36:53.186990 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.186945 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12c512c-b01d-4573-961f-42eb5858bccc" containerName="manager" Jul 07 04:36:53.186990 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.186959 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12c512c-b01d-4573-961f-42eb5858bccc" containerName="manager" Jul 07 04:36:53.187063 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.187030 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12c512c-b01d-4573-961f-42eb5858bccc" containerName="manager" Jul 07 04:36:53.191134 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.191117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.194279 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.194259 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 04:36:53.194365 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.194309 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 04:36:53.194428 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.194372 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cpsxj\"" Jul 07 04:36:53.199080 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.199058 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-ll7cz"] Jul 07 04:36:53.368191 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.368158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-webhook-cert\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.368330 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.368199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvmw\" (UniqueName: \"kubernetes.io/projected/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-kube-api-access-wmvmw\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.469027 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.468954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvmw\" (UniqueName: \"kubernetes.io/projected/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-kube-api-access-wmvmw\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.469152 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.469066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-webhook-cert\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.471433 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.471408 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-webhook-cert\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.495596 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.495573 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvmw\" (UniqueName: \"kubernetes.io/projected/f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4-kube-api-access-wmvmw\") pod \"maas-controller-65fd5f6bc4-ll7cz\" (UID: \"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4\") " pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.502412 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.502383 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:53.528438 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.528413 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12c512c-b01d-4573-961f-42eb5858bccc" path="/var/lib/kubelet/pods/c12c512c-b01d-4573-961f-42eb5858bccc/volumes" Jul 07 04:36:53.635078 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:53.635051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fd5f6bc4-ll7cz"] Jul 07 04:36:53.636734 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:36:53.636709 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a1af7b_1109_4e1b_9ae4_b28d7421bcd4.slice/crio-63d26a2214071087202b82c088bac0e67c1e8036a015fd86679af8cc665aec4b WatchSource:0}: Error finding container 63d26a2214071087202b82c088bac0e67c1e8036a015fd86679af8cc665aec4b: Status 404 returned error can't find the container with id 63d26a2214071087202b82c088bac0e67c1e8036a015fd86679af8cc665aec4b Jul 07 04:36:54.117209 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:54.117179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" event={"ID":"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4","Type":"ContainerStarted","Data":"bf4891c794462ead743a37a120445beb5aae152a18b72feb0c70617ad209665f"} Jul 07 04:36:54.117363 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:54.117256 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" event={"ID":"f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4","Type":"ContainerStarted","Data":"63d26a2214071087202b82c088bac0e67c1e8036a015fd86679af8cc665aec4b"} Jul 07 04:36:54.117363 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:54.117280 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:36:54.133359 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:36:54.133313 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" podStartSLOduration=0.738571492 podStartE2EDuration="1.133298493s" totalImagesPullingTime="394.727001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:36:54.130859774 +0000 UTC m=+787.112793629" watchObservedRunningTime="2026-07-07 04:36:54.133298493 +0000 UTC m=+787.115232360" Jul 07 04:37:05.128454 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:37:05.128421 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65fd5f6bc4-ll7cz" Jul 07 04:38:47.491049 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:38:47.491018 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:38:47.492017 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:38:47.491990 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:43:47.519740 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:43:47.519714 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:43:47.520331 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:43:47.519964 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:45:00.142597 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.142561 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723325-fthkq"] Jul 07 04:45:00.145810 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.145792 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:00.162052 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.162026 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723325-fthkq"] Jul 07 04:45:00.228410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.228376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhdg\" (UniqueName: \"kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg\") pod \"maas-api-key-cleanup-29723325-fthkq\" (UID: \"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa\") " pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:00.329249 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.329210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhdg\" (UniqueName: \"kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg\") pod \"maas-api-key-cleanup-29723325-fthkq\" (UID: \"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa\") " pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:00.338352 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.338322 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhdg\" (UniqueName: \"kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg\") pod \"maas-api-key-cleanup-29723325-fthkq\" (UID: \"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa\") " pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:00.455968 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.455883 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:00.591589 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.591392 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723325-fthkq"] Jul 07 04:45:00.594313 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:45:00.594288 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1a6225_f8b6_43c8_ac3e_da0d582b3faa.slice/crio-b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619 WatchSource:0}: Error finding container b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619: Status 404 returned error can't find the container with id b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619 Jul 07 04:45:00.596214 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:00.596196 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:45:01.394745 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:01.394710 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" event={"ID":"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa","Type":"ContainerStarted","Data":"b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619"} Jul 07 04:45:04.414656 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:04.414618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" event={"ID":"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa","Type":"ContainerStarted","Data":"b22777f1a6ee19a560820a637972f09a7c11d01ee63cb15ae065c17a4d812d2c"} Jul 07 04:45:04.431133 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:04.431082 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" podStartSLOduration=1.5093502650000001 podStartE2EDuration="4.431067196s" totalImagesPullingTime="2.921716931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:45:04.429575858 +0000 UTC m=+1277.411509711" watchObservedRunningTime="2026-07-07 04:45:04.431067196 +0000 UTC m=+1277.413001048" Jul 07 04:45:05.420864 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:05.420824 2563 generic.go:356] "Generic (PLEG): container finished" podID="3c1a6225-f8b6-43c8-ac3e-da0d582b3faa" containerID="b22777f1a6ee19a560820a637972f09a7c11d01ee63cb15ae065c17a4d812d2c" exitCode=0 Jul 07 04:45:05.421253 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:05.420911 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" event={"ID":"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa","Type":"ContainerDied","Data":"b22777f1a6ee19a560820a637972f09a7c11d01ee63cb15ae065c17a4d812d2c"} Jul 07 04:45:06.557608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:06.557584 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:06.578539 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:06.578512 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vhdg\" (UniqueName: \"kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg\") pod \"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa\" (UID: \"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa\") " Jul 07 04:45:06.580617 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:06.580589 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg" (OuterVolumeSpecName: "kube-api-access-7vhdg") pod "3c1a6225-f8b6-43c8-ac3e-da0d582b3faa" (UID: "3c1a6225-f8b6-43c8-ac3e-da0d582b3faa"). InnerVolumeSpecName "kube-api-access-7vhdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:45:06.680199 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:06.680143 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhdg\" (UniqueName: \"kubernetes.io/projected/3c1a6225-f8b6-43c8-ac3e-da0d582b3faa-kube-api-access-7vhdg\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:45:07.431267 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:07.431235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" Jul 07 04:45:07.431433 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:07.431235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723325-fthkq" event={"ID":"3c1a6225-f8b6-43c8-ac3e-da0d582b3faa","Type":"ContainerDied","Data":"b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619"} Jul 07 04:45:07.431433 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:45:07.431352 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9943b640c1e946f7c91ceee14a452f0cf21380d4f712f4b7ce01680e671f619" Jul 07 04:48:44.934310 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.934270 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:44.934908 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.934887 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1a6225-f8b6-43c8-ac3e-da0d582b3faa" containerName="cleanup" Jul 07 04:48:44.934968 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.934914 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1a6225-f8b6-43c8-ac3e-da0d582b3faa" containerName="cleanup" Jul 07 04:48:44.935028 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.935016 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1a6225-f8b6-43c8-ac3e-da0d582b3faa" containerName="cleanup" Jul 07 04:48:44.938315 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.938294 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:44.940516 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.940495 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-b11dd824-serving-cert\"" Jul 07 04:48:44.946148 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:44.945569 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:45.028502 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.028435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.028669 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.028588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.028669 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.028631 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5x5\" (UniqueName: \"kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.129624 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.129589 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.129624 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.129628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5x5\" (UniqueName: \"kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.129895 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.129788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.130457 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.130425 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.132232 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.132201 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.137809 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.137786 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5x5\" (UniqueName: \"kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5\") pod \"maas-api-e2e-ait-b11dd824-75774d578-ndlxv\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.250249 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.250219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:45.390171 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.390147 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:45.392548 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:48:45.392510 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c927dd5_6b37_4012_9d57_e6dcfcf47acf.slice/crio-3591170ec2bbcc46ad9f0efbf1c9e818df174775018b7a49b99ddf96d53d5229 WatchSource:0}: Error finding container 3591170ec2bbcc46ad9f0efbf1c9e818df174775018b7a49b99ddf96d53d5229: Status 404 returned error can't find the container with id 3591170ec2bbcc46ad9f0efbf1c9e818df174775018b7a49b99ddf96d53d5229 Jul 07 04:48:45.461778 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:45.461741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" event={"ID":"3c927dd5-6b37-4012-9d57-e6dcfcf47acf","Type":"ContainerStarted","Data":"3591170ec2bbcc46ad9f0efbf1c9e818df174775018b7a49b99ddf96d53d5229"} Jul 07 04:48:46.467938 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:46.467897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" event={"ID":"3c927dd5-6b37-4012-9d57-e6dcfcf47acf","Type":"ContainerStarted","Data":"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece"} Jul 07 04:48:46.468372 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:46.467969 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:46.488830 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:46.488782 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" podStartSLOduration=2.090563341 podStartE2EDuration="2.488752351s" totalImagesPullingTime="398.18901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:48:46.486964332 +0000 UTC m=+1499.468898183" watchObservedRunningTime="2026-07-07 04:48:46.488752351 +0000 UTC m=+1499.470686202" Jul 07 04:48:47.557047 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:47.556968 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:48:47.557440 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:47.557109 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:48:50.489903 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.489873 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:50.490293 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.490070 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" podUID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" containerName="maas-api" containerID="cri-o://bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece" gracePeriod=30 Jul 07 04:48:50.495027 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.495008 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:50.740509 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.740451 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:50.879175 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.879141 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg5x5\" (UniqueName: \"kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5\") pod \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " Jul 07 04:48:50.879352 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.879225 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls\") pod \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " Jul 07 04:48:50.879352 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.879310 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca\") pod \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\" (UID: \"3c927dd5-6b37-4012-9d57-e6dcfcf47acf\") " Jul 07 04:48:50.879656 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.879631 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c927dd5-6b37-4012-9d57-e6dcfcf47acf" (UID: "3c927dd5-6b37-4012-9d57-e6dcfcf47acf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:48:50.881367 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.881339 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c927dd5-6b37-4012-9d57-e6dcfcf47acf" (UID: "3c927dd5-6b37-4012-9d57-e6dcfcf47acf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:48:50.881469 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.881379 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5" (OuterVolumeSpecName: "kube-api-access-gg5x5") pod "3c927dd5-6b37-4012-9d57-e6dcfcf47acf" (UID: "3c927dd5-6b37-4012-9d57-e6dcfcf47acf"). InnerVolumeSpecName "kube-api-access-gg5x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:48:50.980010 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.979986 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5x5\" (UniqueName: \"kubernetes.io/projected/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-kube-api-access-gg5x5\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:48:50.980010 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.980006 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:48:50.980149 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:50.980016 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c927dd5-6b37-4012-9d57-e6dcfcf47acf-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:48:51.493339 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.493305 2563 generic.go:356] "Generic (PLEG): container finished" podID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" containerID="bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece" exitCode=0 Jul 07 04:48:51.493339 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.493344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" event={"ID":"3c927dd5-6b37-4012-9d57-e6dcfcf47acf","Type":"ContainerDied","Data":"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece"} Jul 07 04:48:51.493837 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.493368 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" Jul 07 04:48:51.493837 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.493407 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv" event={"ID":"3c927dd5-6b37-4012-9d57-e6dcfcf47acf","Type":"ContainerDied","Data":"3591170ec2bbcc46ad9f0efbf1c9e818df174775018b7a49b99ddf96d53d5229"} Jul 07 04:48:51.493837 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.493431 2563 scope.go:117] "RemoveContainer" containerID="bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece" Jul 07 04:48:51.502875 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.502858 2563 scope.go:117] "RemoveContainer" containerID="bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece" Jul 07 04:48:51.503128 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:48:51.503108 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece\": container with ID starting with bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece not found: ID does not exist" containerID="bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece" Jul 07 04:48:51.503211 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.503136 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece"} err="failed to get container status \"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece\": rpc error: code = NotFound desc = could not find container \"bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece\": container with ID starting with bd9b3ab8f7311d071e251fcacdc795e8b2a3b16a54250770b54cac07011b8ece not found: ID does not exist" Jul 07 04:48:51.515277 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.515249 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:51.516960 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.516935 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-b11dd824-75774d578-ndlxv"] Jul 07 04:48:51.527585 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:51.527566 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" path="/var/lib/kubelet/pods/3c927dd5-6b37-4012-9d57-e6dcfcf47acf/volumes" Jul 07 04:48:57.467563 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.467527 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:48:57.468131 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.468111 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" containerName="maas-api" Jul 07 04:48:57.468131 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.468131 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" containerName="maas-api" Jul 07 04:48:57.468273 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.468221 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c927dd5-6b37-4012-9d57-e6dcfcf47acf" containerName="maas-api" Jul 07 04:48:57.473072 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.473048 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.475656 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.475629 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5bdca307-serving-cert\"" Jul 07 04:48:57.482642 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.482599 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:48:57.637483 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.637437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.637663 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.637511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.637663 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.637548 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjct\" (UniqueName: \"kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.738240 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.738216 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.738389 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.738249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjct\" (UniqueName: \"kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.738389 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.738293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.738952 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.738928 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.740655 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.740632 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.747018 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.746996 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjct\" (UniqueName: \"kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct\") pod \"maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.786461 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.786440 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:57.933966 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:57.933937 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:48:57.935481 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:48:57.935435 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dea01f_ec8f_441e_a0fe_7c08698f0070.slice/crio-c510c91605f8d9f3d2c6aa8d82817fe2bd97108b717cea26e9a7aba86270ea25 WatchSource:0}: Error finding container c510c91605f8d9f3d2c6aa8d82817fe2bd97108b717cea26e9a7aba86270ea25: Status 404 returned error can't find the container with id c510c91605f8d9f3d2c6aa8d82817fe2bd97108b717cea26e9a7aba86270ea25 Jul 07 04:48:58.532430 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:58.532345 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" event={"ID":"68dea01f-ec8f-441e-a0fe-7c08698f0070","Type":"ContainerStarted","Data":"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a"} Jul 07 04:48:58.532430 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:58.532406 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:48:58.532880 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:58.532431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" event={"ID":"68dea01f-ec8f-441e-a0fe-7c08698f0070","Type":"ContainerStarted","Data":"c510c91605f8d9f3d2c6aa8d82817fe2bd97108b717cea26e9a7aba86270ea25"} Jul 07 04:48:58.552927 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:48:58.552877 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" podStartSLOduration=1.248233231 podStartE2EDuration="1.552859813s" totalImagesPullingTime="304.626582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:48:58.550804193 +0000 UTC m=+1511.532738044" watchObservedRunningTime="2026-07-07 04:48:58.552859813 +0000 UTC m=+1511.534793665" Jul 07 04:49:03.051642 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.051602 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:49:03.052122 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.051878 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" podUID="68dea01f-ec8f-441e-a0fe-7c08698f0070" containerName="maas-api" containerID="cri-o://8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a" gracePeriod=30 Jul 07 04:49:03.058672 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.058648 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:49:03.305110 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.305054 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:49:03.495577 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.495546 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdjct\" (UniqueName: \"kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct\") pod \"68dea01f-ec8f-441e-a0fe-7c08698f0070\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " Jul 07 04:49:03.495758 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.495667 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls\") pod \"68dea01f-ec8f-441e-a0fe-7c08698f0070\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " Jul 07 04:49:03.495758 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.495700 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca\") pod \"68dea01f-ec8f-441e-a0fe-7c08698f0070\" (UID: \"68dea01f-ec8f-441e-a0fe-7c08698f0070\") " Jul 07 04:49:03.496050 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.496026 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68dea01f-ec8f-441e-a0fe-7c08698f0070" (UID: "68dea01f-ec8f-441e-a0fe-7c08698f0070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:49:03.497568 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.497547 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct" (OuterVolumeSpecName: "kube-api-access-kdjct") pod "68dea01f-ec8f-441e-a0fe-7c08698f0070" (UID: "68dea01f-ec8f-441e-a0fe-7c08698f0070"). InnerVolumeSpecName "kube-api-access-kdjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:49:03.497780 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.497747 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68dea01f-ec8f-441e-a0fe-7c08698f0070" (UID: "68dea01f-ec8f-441e-a0fe-7c08698f0070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:49:03.555386 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.555331 2563 generic.go:356] "Generic (PLEG): container finished" podID="68dea01f-ec8f-441e-a0fe-7c08698f0070" containerID="8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a" exitCode=0 Jul 07 04:49:03.555386 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.555370 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" Jul 07 04:49:03.555505 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.555409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" event={"ID":"68dea01f-ec8f-441e-a0fe-7c08698f0070","Type":"ContainerDied","Data":"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a"} Jul 07 04:49:03.555505 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.555457 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf" event={"ID":"68dea01f-ec8f-441e-a0fe-7c08698f0070","Type":"ContainerDied","Data":"c510c91605f8d9f3d2c6aa8d82817fe2bd97108b717cea26e9a7aba86270ea25"} Jul 07 04:49:03.555505 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.555483 2563 scope.go:117] "RemoveContainer" containerID="8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a" Jul 07 04:49:03.564007 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.563993 2563 scope.go:117] "RemoveContainer" containerID="8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a" Jul 07 04:49:03.564218 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:49:03.564202 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a\": container with ID starting with 8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a not found: ID does not exist" containerID="8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a" Jul 07 04:49:03.564259 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.564226 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a"} err="failed to get container status \"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a\": rpc error: code = NotFound desc = could not find container \"8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a\": container with ID starting with 8a48ae3b06e47a76615534a1b23df488bca4c8b4f15e630ee3fffc962329876a not found: ID does not exist" Jul 07 04:49:03.585491 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.585472 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:49:03.596297 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.596275 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68dea01f-ec8f-441e-a0fe-7c08698f0070-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:03.596365 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.596297 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dea01f-ec8f-441e-a0fe-7c08698f0070-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:03.596365 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.596307 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjct\" (UniqueName: \"kubernetes.io/projected/68dea01f-ec8f-441e-a0fe-7c08698f0070-kube-api-access-kdjct\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:03.608403 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:03.608379 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5bdca307-757d9947bf-d4sdf"] Jul 07 04:49:05.527519 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:05.527486 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68dea01f-ec8f-441e-a0fe-7c08698f0070" path="/var/lib/kubelet/pods/68dea01f-ec8f-441e-a0fe-7c08698f0070/volumes" Jul 07 04:49:16.174368 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.174332 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:16.174952 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.174934 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68dea01f-ec8f-441e-a0fe-7c08698f0070" containerName="maas-api" Jul 07 04:49:16.174998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.174956 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68dea01f-ec8f-441e-a0fe-7c08698f0070" containerName="maas-api" Jul 07 04:49:16.175084 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.175072 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68dea01f-ec8f-441e-a0fe-7c08698f0070" containerName="maas-api" Jul 07 04:49:16.179341 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.179317 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.182037 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.182012 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-93ca8cbb-serving-cert\"" Jul 07 04:49:16.192543 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.192523 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:16.192709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.192685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.192846 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.192830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.192914 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.192869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzk8\" (UniqueName: \"kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.293723 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.293695 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.293877 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.293751 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.293877 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.293799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzk8\" (UniqueName: \"kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.294311 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.294289 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.296231 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.296208 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.306316 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.306290 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzk8\" (UniqueName: \"kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8\") pod \"maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.492244 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.492211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:16.634230 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:16.634205 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:16.636822 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:49:16.636792 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385f457e_400d_4f30_91e3_3dabef16fa76.slice/crio-46126c7bae80564bc28e089c59acf27ba6708ec792eba6e7f30dd6bcff16a181 WatchSource:0}: Error finding container 46126c7bae80564bc28e089c59acf27ba6708ec792eba6e7f30dd6bcff16a181: Status 404 returned error can't find the container with id 46126c7bae80564bc28e089c59acf27ba6708ec792eba6e7f30dd6bcff16a181 Jul 07 04:49:17.628446 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:17.628411 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" event={"ID":"385f457e-400d-4f30-91e3-3dabef16fa76","Type":"ContainerStarted","Data":"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d"} Jul 07 04:49:17.628833 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:17.628461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" event={"ID":"385f457e-400d-4f30-91e3-3dabef16fa76","Type":"ContainerStarted","Data":"46126c7bae80564bc28e089c59acf27ba6708ec792eba6e7f30dd6bcff16a181"} Jul 07 04:49:17.628833 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:17.628504 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:17.646850 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:17.646801 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" podStartSLOduration=1.304242789 podStartE2EDuration="1.646786955s" totalImagesPullingTime="342.544166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:49:17.645479407 +0000 UTC m=+1530.627413286" watchObservedRunningTime="2026-07-07 04:49:17.646786955 +0000 UTC m=+1530.628720808" Jul 07 04:49:20.894812 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:20.894757 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:20.895274 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:20.895040 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" podUID="385f457e-400d-4f30-91e3-3dabef16fa76" containerName="maas-api" containerID="cri-o://34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d" gracePeriod=30 Jul 07 04:49:21.143085 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.143056 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:21.229999 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.229927 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca\") pod \"385f457e-400d-4f30-91e3-3dabef16fa76\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " Jul 07 04:49:21.229999 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.229966 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnzk8\" (UniqueName: \"kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8\") pod \"385f457e-400d-4f30-91e3-3dabef16fa76\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " Jul 07 04:49:21.230189 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.230009 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls\") pod \"385f457e-400d-4f30-91e3-3dabef16fa76\" (UID: \"385f457e-400d-4f30-91e3-3dabef16fa76\") " Jul 07 04:49:21.230347 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.230320 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "385f457e-400d-4f30-91e3-3dabef16fa76" (UID: "385f457e-400d-4f30-91e3-3dabef16fa76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:49:21.232158 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.232128 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8" (OuterVolumeSpecName: "kube-api-access-hnzk8") pod "385f457e-400d-4f30-91e3-3dabef16fa76" (UID: "385f457e-400d-4f30-91e3-3dabef16fa76"). InnerVolumeSpecName "kube-api-access-hnzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:49:21.232275 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.232170 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "385f457e-400d-4f30-91e3-3dabef16fa76" (UID: "385f457e-400d-4f30-91e3-3dabef16fa76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:49:21.331336 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.331312 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/385f457e-400d-4f30-91e3-3dabef16fa76-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:21.331336 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.331334 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzk8\" (UniqueName: \"kubernetes.io/projected/385f457e-400d-4f30-91e3-3dabef16fa76-kube-api-access-hnzk8\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:21.331465 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.331345 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/385f457e-400d-4f30-91e3-3dabef16fa76-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:49:21.648874 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.648840 2563 generic.go:356] "Generic (PLEG): container finished" podID="385f457e-400d-4f30-91e3-3dabef16fa76" containerID="34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d" exitCode=0 Jul 07 04:49:21.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.648887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" event={"ID":"385f457e-400d-4f30-91e3-3dabef16fa76","Type":"ContainerDied","Data":"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d"} Jul 07 04:49:21.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.648894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" Jul 07 04:49:21.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.648941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd" event={"ID":"385f457e-400d-4f30-91e3-3dabef16fa76","Type":"ContainerDied","Data":"46126c7bae80564bc28e089c59acf27ba6708ec792eba6e7f30dd6bcff16a181"} Jul 07 04:49:21.648998 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.648973 2563 scope.go:117] "RemoveContainer" containerID="34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d" Jul 07 04:49:21.657579 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.657560 2563 scope.go:117] "RemoveContainer" containerID="34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d" Jul 07 04:49:21.657857 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:49:21.657837 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d\": container with ID starting with 34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d not found: ID does not exist" containerID="34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d" Jul 07 04:49:21.657939 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.657870 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d"} err="failed to get container status \"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d\": rpc error: code = NotFound desc = could not find container \"34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d\": container with ID starting with 34de6470bb52007dd8c86fc11fa2325061e8a6c62445271a645ef958600a2e3d not found: ID does not exist" Jul 07 04:49:21.668858 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.668837 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:21.672540 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:21.672520 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-93ca8cbb-75f659b7d8-8fhqd"] Jul 07 04:49:23.528001 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:49:23.527968 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385f457e-400d-4f30-91e3-3dabef16fa76" path="/var/lib/kubelet/pods/385f457e-400d-4f30-91e3-3dabef16fa76/volumes" Jul 07 04:51:05.282049 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.282006 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:05.282600 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.282555 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385f457e-400d-4f30-91e3-3dabef16fa76" containerName="maas-api" Jul 07 04:51:05.282600 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.282571 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="385f457e-400d-4f30-91e3-3dabef16fa76" containerName="maas-api" Jul 07 04:51:05.282698 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.282685 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="385f457e-400d-4f30-91e3-3dabef16fa76" containerName="maas-api" Jul 07 04:51:05.286194 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.286166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.288793 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.288747 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-dbd27213-serving-cert\"" Jul 07 04:51:05.295858 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.295834 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:05.338606 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.338575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.338734 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.338651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.338734 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.338692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42mrw\" (UniqueName: \"kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.439845 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.439810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.440019 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.439877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.440019 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.439929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42mrw\" (UniqueName: \"kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.440019 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:51:05.439983 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-dbd27213-serving-cert: secret "maas-api-e2e-mt-dbd27213-serving-cert" not found Jul 07 04:51:05.440151 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:51:05.440060 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls podName:32678f1a-8ca3-478b-8498-19daec4edaf7 nodeName:}" failed. No retries permitted until 2026-07-07 04:51:05.940031977 +0000 UTC m=+1638.921965808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls") pod "maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" (UID: "32678f1a-8ca3-478b-8498-19daec4edaf7") : secret "maas-api-e2e-mt-dbd27213-serving-cert" not found Jul 07 04:51:05.440585 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.440560 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.449147 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.449124 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42mrw\" (UniqueName: \"kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.944749 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.944711 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:05.947094 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:05.947074 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") pod \"maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:06.200007 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:06.199926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:06.359947 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:06.359913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:06.363330 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:51:06.363294 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32678f1a_8ca3_478b_8498_19daec4edaf7.slice/crio-ddb80be1ae92d761a17578f859bc6600f0130337da05d9eb21cad442ec67cfdc WatchSource:0}: Error finding container ddb80be1ae92d761a17578f859bc6600f0130337da05d9eb21cad442ec67cfdc: Status 404 returned error can't find the container with id ddb80be1ae92d761a17578f859bc6600f0130337da05d9eb21cad442ec67cfdc Jul 07 04:51:06.364558 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:06.364540 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:51:07.137209 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:07.137128 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" event={"ID":"32678f1a-8ca3-478b-8498-19daec4edaf7","Type":"ContainerStarted","Data":"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5"} Jul 07 04:51:07.137335 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:07.137215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" event={"ID":"32678f1a-8ca3-478b-8498-19daec4edaf7","Type":"ContainerStarted","Data":"ddb80be1ae92d761a17578f859bc6600f0130337da05d9eb21cad442ec67cfdc"} Jul 07 04:51:07.137381 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:07.137332 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:07.156777 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:07.156700 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" podStartSLOduration=1.6925005899999999 podStartE2EDuration="2.156687662s" totalImagesPullingTime="464.187072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:51:07.15564537 +0000 UTC m=+1640.137579223" watchObservedRunningTime="2026-07-07 04:51:07.156687662 +0000 UTC m=+1640.138621514" Jul 07 04:51:13.147184 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:13.147150 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:17.490692 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.490656 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:17.491180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.490946 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" podUID="32678f1a-8ca3-478b-8498-19daec4edaf7" containerName="maas-api" containerID="cri-o://4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5" gracePeriod=30 Jul 07 04:51:17.737193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.737170 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:17.843526 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.843461 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") pod \"32678f1a-8ca3-478b-8498-19daec4edaf7\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " Jul 07 04:51:17.843526 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.843491 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42mrw\" (UniqueName: \"kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw\") pod \"32678f1a-8ca3-478b-8498-19daec4edaf7\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " Jul 07 04:51:17.843720 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.843559 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca\") pod \"32678f1a-8ca3-478b-8498-19daec4edaf7\" (UID: \"32678f1a-8ca3-478b-8498-19daec4edaf7\") " Jul 07 04:51:17.843994 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.843971 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32678f1a-8ca3-478b-8498-19daec4edaf7" (UID: "32678f1a-8ca3-478b-8498-19daec4edaf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:51:17.845535 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.845514 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32678f1a-8ca3-478b-8498-19daec4edaf7" (UID: "32678f1a-8ca3-478b-8498-19daec4edaf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:51:17.845634 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.845611 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw" (OuterVolumeSpecName: "kube-api-access-42mrw") pod "32678f1a-8ca3-478b-8498-19daec4edaf7" (UID: "32678f1a-8ca3-478b-8498-19daec4edaf7"). InnerVolumeSpecName "kube-api-access-42mrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:51:17.944973 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.944952 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32678f1a-8ca3-478b-8498-19daec4edaf7-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:51:17.944973 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.944972 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42mrw\" (UniqueName: \"kubernetes.io/projected/32678f1a-8ca3-478b-8498-19daec4edaf7-kube-api-access-42mrw\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:51:17.945102 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:17.944984 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32678f1a-8ca3-478b-8498-19daec4edaf7-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:51:18.197633 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.197552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" event={"ID":"32678f1a-8ca3-478b-8498-19daec4edaf7","Type":"ContainerDied","Data":"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5"} Jul 07 04:51:18.197633 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.197590 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" Jul 07 04:51:18.197633 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.197557 2563 generic.go:356] "Generic (PLEG): container finished" podID="32678f1a-8ca3-478b-8498-19daec4edaf7" containerID="4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5" exitCode=0 Jul 07 04:51:18.197903 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.197650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd" event={"ID":"32678f1a-8ca3-478b-8498-19daec4edaf7","Type":"ContainerDied","Data":"ddb80be1ae92d761a17578f859bc6600f0130337da05d9eb21cad442ec67cfdc"} Jul 07 04:51:18.197903 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.197652 2563 scope.go:117] "RemoveContainer" containerID="4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5" Jul 07 04:51:18.207506 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.207485 2563 scope.go:117] "RemoveContainer" containerID="4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5" Jul 07 04:51:18.207737 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:51:18.207718 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5\": container with ID starting with 4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5 not found: ID does not exist" containerID="4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5" Jul 07 04:51:18.207821 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.207744 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5"} err="failed to get container status \"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5\": rpc error: code = NotFound desc = could not find container \"4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5\": container with ID starting with 4fee803d3e5e6932806b932e179c1400fe2aed226aa2f2ffc66bb3d572bdc6d5 not found: ID does not exist" Jul 07 04:51:18.220085 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.220063 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:18.225982 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:18.225959 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-dbd27213-57c6dc74b6-zwcgd"] Jul 07 04:51:19.527787 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:19.527716 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32678f1a-8ca3-478b-8498-19daec4edaf7" path="/var/lib/kubelet/pods/32678f1a-8ca3-478b-8498-19daec4edaf7/volumes" Jul 07 04:51:46.521155 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.521124 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:51:46.521626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.521522 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32678f1a-8ca3-478b-8498-19daec4edaf7" containerName="maas-api" Jul 07 04:51:46.521626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.521534 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32678f1a-8ca3-478b-8498-19daec4edaf7" containerName="maas-api" Jul 07 04:51:46.521626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.521591 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32678f1a-8ca3-478b-8498-19daec4edaf7" containerName="maas-api" Jul 07 04:51:46.524932 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.524912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.527848 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.527830 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-036fb253-serving-cert\"" Jul 07 04:51:46.537978 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.537947 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:51:46.576092 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.576061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.576222 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.576197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfc2d\" (UniqueName: \"kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.576294 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.576259 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.677231 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.677204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfc2d\" (UniqueName: \"kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.677361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.677241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.677361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.677275 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.677456 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:51:46.677382 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-036fb253-serving-cert: secret "maas-api-e2e-mt-036fb253-serving-cert" not found Jul 07 04:51:46.677492 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:51:46.677463 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls podName:3394bf0c-cda1-4654-ac23-550d1562ca93 nodeName:}" failed. No retries permitted until 2026-07-07 04:51:47.177442141 +0000 UTC m=+1680.159375975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls") pod "maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" (UID: "3394bf0c-cda1-4654-ac23-550d1562ca93") : secret "maas-api-e2e-mt-036fb253-serving-cert" not found Jul 07 04:51:46.677947 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.677929 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:46.686121 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:46.686100 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfc2d\" (UniqueName: \"kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:47.180063 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:47.180027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:47.182936 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:47.182911 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") pod \"maas-api-e2e-mt-036fb253-775b7bd794-g2vfm\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:47.436557 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:47.436472 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:47.582000 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:47.581978 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:51:47.584130 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:51:47.584105 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3394bf0c_cda1_4654_ac23_550d1562ca93.slice/crio-aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c WatchSource:0}: Error finding container aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c: Status 404 returned error can't find the container with id aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c Jul 07 04:51:48.350891 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:48.350858 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" event={"ID":"3394bf0c-cda1-4654-ac23-550d1562ca93","Type":"ContainerStarted","Data":"215408116d9ae8e82fc403d50ec18297058468f87f3c2358b8fee39659a64261"} Jul 07 04:51:48.351072 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:48.350910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" event={"ID":"3394bf0c-cda1-4654-ac23-550d1562ca93","Type":"ContainerStarted","Data":"aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c"} Jul 07 04:51:48.351072 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:48.351007 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:51:48.371581 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:48.371533 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" podStartSLOduration=1.991020663 podStartE2EDuration="2.371520457s" totalImagesPullingTime="380.499794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:51:48.369484718 +0000 UTC m=+1681.351418571" watchObservedRunningTime="2026-07-07 04:51:48.371520457 +0000 UTC m=+1681.353454309" Jul 07 04:51:54.360997 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:51:54.360969 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:52:39.353927 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.353879 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:52:39.354885 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.354125 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" podUID="3394bf0c-cda1-4654-ac23-550d1562ca93" containerName="maas-api" containerID="cri-o://215408116d9ae8e82fc403d50ec18297058468f87f3c2358b8fee39659a64261" gracePeriod=30 Jul 07 04:52:39.595698 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.595673 2563 generic.go:356] "Generic (PLEG): container finished" podID="3394bf0c-cda1-4654-ac23-550d1562ca93" containerID="215408116d9ae8e82fc403d50ec18297058468f87f3c2358b8fee39659a64261" exitCode=0 Jul 07 04:52:39.595817 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.595710 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" event={"ID":"3394bf0c-cda1-4654-ac23-550d1562ca93","Type":"ContainerDied","Data":"215408116d9ae8e82fc403d50ec18297058468f87f3c2358b8fee39659a64261"} Jul 07 04:52:39.595817 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.595747 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" event={"ID":"3394bf0c-cda1-4654-ac23-550d1562ca93","Type":"ContainerDied","Data":"aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c"} Jul 07 04:52:39.595817 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.595762 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae1ee325a55d9685d3d2dad9fb793dcbf34da057c5a7db198ae18012d53318c" Jul 07 04:52:39.608423 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.608378 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:52:39.695730 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.695705 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfc2d\" (UniqueName: \"kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d\") pod \"3394bf0c-cda1-4654-ac23-550d1562ca93\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " Jul 07 04:52:39.695881 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.695825 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") pod \"3394bf0c-cda1-4654-ac23-550d1562ca93\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " Jul 07 04:52:39.695881 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.695849 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca\") pod \"3394bf0c-cda1-4654-ac23-550d1562ca93\" (UID: \"3394bf0c-cda1-4654-ac23-550d1562ca93\") " Jul 07 04:52:39.696233 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.696210 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3394bf0c-cda1-4654-ac23-550d1562ca93" (UID: "3394bf0c-cda1-4654-ac23-550d1562ca93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:52:39.697672 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.697656 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d" (OuterVolumeSpecName: "kube-api-access-zfc2d") pod "3394bf0c-cda1-4654-ac23-550d1562ca93" (UID: "3394bf0c-cda1-4654-ac23-550d1562ca93"). InnerVolumeSpecName "kube-api-access-zfc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:52:39.697864 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.697844 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3394bf0c-cda1-4654-ac23-550d1562ca93" (UID: "3394bf0c-cda1-4654-ac23-550d1562ca93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:52:39.796404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.796377 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3394bf0c-cda1-4654-ac23-550d1562ca93-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:52:39.796404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.796402 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3394bf0c-cda1-4654-ac23-550d1562ca93-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:52:39.796543 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:39.796411 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfc2d\" (UniqueName: \"kubernetes.io/projected/3394bf0c-cda1-4654-ac23-550d1562ca93-kube-api-access-zfc2d\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:52:40.600991 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:40.600960 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm" Jul 07 04:52:40.622386 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:40.622361 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:52:40.626061 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:40.626032 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-036fb253-775b7bd794-g2vfm"] Jul 07 04:52:41.527996 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:52:41.527962 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3394bf0c-cda1-4654-ac23-550d1562ca93" path="/var/lib/kubelet/pods/3394bf0c-cda1-4654-ac23-550d1562ca93/volumes" Jul 07 04:53:47.584118 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:53:47.584087 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:53:47.584902 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:53:47.584884 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:54:30.494179 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.494137 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:54:30.494626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.494510 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3394bf0c-cda1-4654-ac23-550d1562ca93" containerName="maas-api" Jul 07 04:54:30.494626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.494523 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3394bf0c-cda1-4654-ac23-550d1562ca93" containerName="maas-api" Jul 07 04:54:30.494626 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.494574 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3394bf0c-cda1-4654-ac23-550d1562ca93" containerName="maas-api" Jul 07 04:54:30.498146 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.498123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.500407 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.500388 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-94ebac-serving-cert\"" Jul 07 04:54:30.508886 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.508862 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:54:30.681112 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.681074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlqv\" (UniqueName: \"kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.681265 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.681140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.681323 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.681279 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.781759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.781688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlqv\" (UniqueName: \"kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.781759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.781719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.781759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.781758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.782112 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:54:30.781906 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-94ebac-serving-cert: secret "maas-api-e2e-inf-a-94ebac-serving-cert" not found Jul 07 04:54:30.782112 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:54:30.782008 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls podName:fa1e25bb-2401-4918-997d-2d8eb53d1038 nodeName:}" failed. No retries permitted until 2026-07-07 04:54:31.28198319 +0000 UTC m=+1844.263917022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls") pod "maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" (UID: "fa1e25bb-2401-4918-997d-2d8eb53d1038") : secret "maas-api-e2e-inf-a-94ebac-serving-cert" not found Jul 07 04:54:30.782341 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.782324 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:30.790052 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:30.790032 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlqv\" (UniqueName: \"kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:31.285382 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:31.285354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:31.287638 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:31.287617 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") pod \"maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:31.410219 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:31.410194 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:31.545736 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:31.545706 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:54:31.546824 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:54:31.546794 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1e25bb_2401_4918_997d_2d8eb53d1038.slice/crio-054f2c4e3fdb8968b67bdb4adc5726dc00e1471baaee3c04d5075ee137656c5d WatchSource:0}: Error finding container 054f2c4e3fdb8968b67bdb4adc5726dc00e1471baaee3c04d5075ee137656c5d: Status 404 returned error can't find the container with id 054f2c4e3fdb8968b67bdb4adc5726dc00e1471baaee3c04d5075ee137656c5d Jul 07 04:54:32.131506 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:32.131473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" event={"ID":"fa1e25bb-2401-4918-997d-2d8eb53d1038","Type":"ContainerStarted","Data":"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d"} Jul 07 04:54:32.131667 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:32.131530 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" event={"ID":"fa1e25bb-2401-4918-997d-2d8eb53d1038","Type":"ContainerStarted","Data":"054f2c4e3fdb8968b67bdb4adc5726dc00e1471baaee3c04d5075ee137656c5d"} Jul 07 04:54:32.131667 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:32.131590 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:32.147934 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:32.147891 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" podStartSLOduration=1.742529673 podStartE2EDuration="2.1478789s" totalImagesPullingTime="405.349227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:54:32.146756185 +0000 UTC m=+1845.128690038" watchObservedRunningTime="2026-07-07 04:54:32.1478789 +0000 UTC m=+1845.129812743" Jul 07 04:54:38.141871 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.141845 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:54:38.375268 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.375233 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:54:38.379410 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.379390 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.381793 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.381740 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-ee0733-serving-cert\"" Jul 07 04:54:38.387851 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.387453 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:54:38.448622 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.448562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktn2\" (UniqueName: \"kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.448622 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.448598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.448749 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.448644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.549120 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.549089 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktn2\" (UniqueName: \"kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.549257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.549133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.549257 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.549164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.549370 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:54:38.549265 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-ee0733-serving-cert: secret "maas-api-e2e-inf-b-ee0733-serving-cert" not found Jul 07 04:54:38.549370 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:54:38.549315 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls podName:8ced7c05-b32e-4fd9-ba56-bf1d71076ecb nodeName:}" failed. No retries permitted until 2026-07-07 04:54:39.049300011 +0000 UTC m=+1852.031233845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls") pod "maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" (UID: "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb") : secret "maas-api-e2e-inf-b-ee0733-serving-cert" not found Jul 07 04:54:38.550066 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.550043 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:38.557320 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:38.557289 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktn2\" (UniqueName: \"kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:39.055221 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:39.055178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:39.057513 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:39.057490 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:39.291816 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:39.291763 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:39.427065 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:39.427036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:54:39.430048 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:54:39.430018 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ced7c05_b32e_4fd9_ba56_bf1d71076ecb.slice/crio-71e6bb46db7fcc7fbd01c8d814d5f2280cada077bd49df02a9392968206406fb WatchSource:0}: Error finding container 71e6bb46db7fcc7fbd01c8d814d5f2280cada077bd49df02a9392968206406fb: Status 404 returned error can't find the container with id 71e6bb46db7fcc7fbd01c8d814d5f2280cada077bd49df02a9392968206406fb Jul 07 04:54:40.175164 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:40.175064 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" event={"ID":"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb","Type":"ContainerStarted","Data":"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d"} Jul 07 04:54:40.175164 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:40.175132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" event={"ID":"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb","Type":"ContainerStarted","Data":"71e6bb46db7fcc7fbd01c8d814d5f2280cada077bd49df02a9392968206406fb"} Jul 07 04:54:40.175371 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:40.175176 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:54:40.193521 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:40.193473 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" podStartSLOduration=1.798302252 podStartE2EDuration="2.193458075s" totalImagesPullingTime="395.155823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:54:40.191556412 +0000 UTC m=+1853.173490266" watchObservedRunningTime="2026-07-07 04:54:40.193458075 +0000 UTC m=+1853.175391927" Jul 07 04:54:46.185546 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:54:46.185513 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:55:38.101073 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.101037 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:55:38.101988 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.101930 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerName="maas-api" containerID="cri-o://9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d" gracePeriod=30 Jul 07 04:55:38.136651 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.136622 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.47:8443/health\": dial tcp 10.132.0.47:8443: connect: connection refused" Jul 07 04:55:38.356714 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.356663 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:55:38.469193 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.469156 2563 generic.go:356] "Generic (PLEG): container finished" podID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerID="9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d" exitCode=0 Jul 07 04:55:38.469328 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.469196 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" Jul 07 04:55:38.469328 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.469224 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" event={"ID":"fa1e25bb-2401-4918-997d-2d8eb53d1038","Type":"ContainerDied","Data":"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d"} Jul 07 04:55:38.469328 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.469271 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8" event={"ID":"fa1e25bb-2401-4918-997d-2d8eb53d1038","Type":"ContainerDied","Data":"054f2c4e3fdb8968b67bdb4adc5726dc00e1471baaee3c04d5075ee137656c5d"} Jul 07 04:55:38.469328 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.469294 2563 scope.go:117] "RemoveContainer" containerID="9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d" Jul 07 04:55:38.477939 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.477912 2563 scope.go:117] "RemoveContainer" containerID="9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d" Jul 07 04:55:38.478207 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:55:38.478185 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d\": container with ID starting with 9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d not found: ID does not exist" containerID="9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d" Jul 07 04:55:38.478306 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.478216 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d"} err="failed to get container status \"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d\": rpc error: code = NotFound desc = could not find container \"9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d\": container with ID starting with 9c1dcd7a416eb5b7e061de22cc05dc31c96e568c970ce2e6bd65e3419bc40c0d not found: ID does not exist" Jul 07 04:55:38.526004 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.525977 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca\") pod \"fa1e25bb-2401-4918-997d-2d8eb53d1038\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " Jul 07 04:55:38.526171 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.526016 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhlqv\" (UniqueName: \"kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv\") pod \"fa1e25bb-2401-4918-997d-2d8eb53d1038\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " Jul 07 04:55:38.526171 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.526046 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") pod \"fa1e25bb-2401-4918-997d-2d8eb53d1038\" (UID: \"fa1e25bb-2401-4918-997d-2d8eb53d1038\") " Jul 07 04:55:38.526353 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.526335 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa1e25bb-2401-4918-997d-2d8eb53d1038" (UID: "fa1e25bb-2401-4918-997d-2d8eb53d1038"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:55:38.528136 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.528109 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa1e25bb-2401-4918-997d-2d8eb53d1038" (UID: "fa1e25bb-2401-4918-997d-2d8eb53d1038"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:55:38.528223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.528140 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv" (OuterVolumeSpecName: "kube-api-access-hhlqv") pod "fa1e25bb-2401-4918-997d-2d8eb53d1038" (UID: "fa1e25bb-2401-4918-997d-2d8eb53d1038"). InnerVolumeSpecName "kube-api-access-hhlqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:55:38.626726 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.626667 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e25bb-2401-4918-997d-2d8eb53d1038-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:55:38.626726 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.626692 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlqv\" (UniqueName: \"kubernetes.io/projected/fa1e25bb-2401-4918-997d-2d8eb53d1038-kube-api-access-hhlqv\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:55:38.626726 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.626707 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1e25bb-2401-4918-997d-2d8eb53d1038-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:55:38.790059 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.790032 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:55:38.794241 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:38.794222 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-94ebac-789b9bf657-jqvr8"] Jul 07 04:55:39.527434 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:55:39.527402 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" path="/var/lib/kubelet/pods/fa1e25bb-2401-4918-997d-2d8eb53d1038/volumes" Jul 07 04:56:20.956176 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:20.956143 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:56:20.956704 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:20.956343 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerName="maas-api" containerID="cri-o://0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d" gracePeriod=30 Jul 07 04:56:21.219762 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.219712 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:56:21.272227 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.272205 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") pod \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " Jul 07 04:56:21.274202 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.274180 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" (UID: "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:56:21.372982 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.372956 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca\") pod \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " Jul 07 04:56:21.373088 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.373065 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pktn2\" (UniqueName: \"kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2\") pod \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\" (UID: \"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb\") " Jul 07 04:56:21.373212 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.373200 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:21.373279 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.373258 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" (UID: "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:56:21.375018 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.375002 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2" (OuterVolumeSpecName: "kube-api-access-pktn2") pod "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" (UID: "8ced7c05-b32e-4fd9-ba56-bf1d71076ecb"). InnerVolumeSpecName "kube-api-access-pktn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:56:21.473754 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.473696 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:21.473754 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.473720 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktn2\" (UniqueName: \"kubernetes.io/projected/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb-kube-api-access-pktn2\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:21.685072 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.685045 2563 generic.go:356] "Generic (PLEG): container finished" podID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerID="0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d" exitCode=0 Jul 07 04:56:21.685180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.685096 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" Jul 07 04:56:21.685180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.685132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" event={"ID":"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb","Type":"ContainerDied","Data":"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d"} Jul 07 04:56:21.685256 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.685193 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" event={"ID":"8ced7c05-b32e-4fd9-ba56-bf1d71076ecb","Type":"ContainerDied","Data":"71e6bb46db7fcc7fbd01c8d814d5f2280cada077bd49df02a9392968206406fb"} Jul 07 04:56:21.685256 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.685226 2563 scope.go:117] "RemoveContainer" containerID="0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d" Jul 07 04:56:21.697217 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.697199 2563 scope.go:117] "RemoveContainer" containerID="0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d" Jul 07 04:56:21.697473 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:21.697453 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d\": container with ID starting with 0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d not found: ID does not exist" containerID="0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d" Jul 07 04:56:21.697563 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.697482 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d"} err="failed to get container status \"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d\": rpc error: code = NotFound desc = could not find container \"0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d\": container with ID starting with 0304d45ed083049e66a46614bc67b082d81a926771c12e5b9cdd7fc87f3cab8d not found: ID does not exist" Jul 07 04:56:21.700481 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.700458 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:56:21.704361 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:21.704332 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j"] Jul 07 04:56:23.529127 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:23.529086 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" path="/var/lib/kubelet/pods/8ced7c05-b32e-4fd9-ba56-bf1d71076ecb/volumes" Jul 07 04:56:24.182307 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:24.182252 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-b-ee0733-7d5c79dd77-wbc8j" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.48:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 07 04:56:30.979190 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979154 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979574 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerName="maas-api" Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979586 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerName="maas-api" Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979597 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerName="maas-api" Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979603 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerName="maas-api" Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979672 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ced7c05-b32e-4fd9-ba56-bf1d71076ecb" containerName="maas-api" Jul 07 04:56:30.979743 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.979683 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1e25bb-2401-4918-997d-2d8eb53d1038" containerName="maas-api" Jul 07 04:56:30.984197 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.984176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:30.987157 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.987134 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-073254-serving-cert\"" Jul 07 04:56:30.993384 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:30.993358 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:31.150332 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.150296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9glb\" (UniqueName: \"kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.150507 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.150341 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.150507 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.150371 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.251070 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.251043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9glb\" (UniqueName: \"kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.251218 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.251076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.251218 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.251095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.251218 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:31.251195 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-073254-serving-cert: secret "maas-api-e2e-api-a-073254-serving-cert" not found Jul 07 04:56:31.251380 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:31.251268 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls podName:c4d97be6-23ec-4a27-b5eb-b0c48a9b2962 nodeName:}" failed. No retries permitted until 2026-07-07 04:56:31.75124737 +0000 UTC m=+1964.733181200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls") pod "maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" (UID: "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962") : secret "maas-api-e2e-api-a-073254-serving-cert" not found Jul 07 04:56:31.251715 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.251697 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.261623 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.261595 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9glb\" (UniqueName: \"kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.755199 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.755163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.758163 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.758135 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") pod \"maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:31.896077 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:31.896030 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:32.048608 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.048544 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:32.051145 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:56:32.051115 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d97be6_23ec_4a27_b5eb_b0c48a9b2962.slice/crio-c279d5c7383ff9a15b0443bb0fa4a7fc7d9182efbd5ae59146621b2e214195f6 WatchSource:0}: Error finding container c279d5c7383ff9a15b0443bb0fa4a7fc7d9182efbd5ae59146621b2e214195f6: Status 404 returned error can't find the container with id c279d5c7383ff9a15b0443bb0fa4a7fc7d9182efbd5ae59146621b2e214195f6 Jul 07 04:56:32.052566 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.052545 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:56:32.746018 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.745984 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" event={"ID":"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962","Type":"ContainerStarted","Data":"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a"} Jul 07 04:56:32.746180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.746049 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:32.746180 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.746068 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" event={"ID":"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962","Type":"ContainerStarted","Data":"c279d5c7383ff9a15b0443bb0fa4a7fc7d9182efbd5ae59146621b2e214195f6"} Jul 07 04:56:32.763346 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:32.763299 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" podStartSLOduration=2.366119755 podStartE2EDuration="2.763284025s" totalImagesPullingTime="397.16427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:56:32.761059521 +0000 UTC m=+1965.742993373" watchObservedRunningTime="2026-07-07 04:56:32.763284025 +0000 UTC m=+1965.745217877" Jul 07 04:56:38.756602 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:38.756567 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:38.980289 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:38.980252 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:56:38.984383 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:38.984360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:38.987397 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:38.987371 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-59dc16-serving-cert\"" Jul 07 04:56:38.997228 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:38.997206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:56:39.114250 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.114165 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.114378 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.114239 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9h2t\" (UniqueName: \"kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.114378 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.114284 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.215433 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.215394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.215595 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.215445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9h2t\" (UniqueName: \"kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.215595 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.215467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.215669 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:39.215626 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-59dc16-serving-cert: secret "maas-api-e2e-api-b-59dc16-serving-cert" not found Jul 07 04:56:39.215703 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:39.215687 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls podName:23020b0b-75ef-4b52-9375-d5c36578aa39 nodeName:}" failed. No retries permitted until 2026-07-07 04:56:39.71566986 +0000 UTC m=+1972.697603689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls") pod "maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" (UID: "23020b0b-75ef-4b52-9375-d5c36578aa39") : secret "maas-api-e2e-api-b-59dc16-serving-cert" not found Jul 07 04:56:39.216208 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.216186 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.224243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.224220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9h2t\" (UniqueName: \"kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.720928 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.720890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.723251 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.723227 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") pod \"maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:39.898970 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:39.898936 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:40.040046 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:40.040007 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:56:40.043824 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:56:40.043798 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23020b0b_75ef_4b52_9375_d5c36578aa39.slice/crio-c1048fcb8918aed1dd1b9cab3e1c960179d73cdd7d4698009fec807dab513f7d WatchSource:0}: Error finding container c1048fcb8918aed1dd1b9cab3e1c960179d73cdd7d4698009fec807dab513f7d: Status 404 returned error can't find the container with id c1048fcb8918aed1dd1b9cab3e1c960179d73cdd7d4698009fec807dab513f7d Jul 07 04:56:40.785962 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:40.785864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" event={"ID":"23020b0b-75ef-4b52-9375-d5c36578aa39","Type":"ContainerStarted","Data":"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f"} Jul 07 04:56:40.785962 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:40.785929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" event={"ID":"23020b0b-75ef-4b52-9375-d5c36578aa39","Type":"ContainerStarted","Data":"c1048fcb8918aed1dd1b9cab3e1c960179d73cdd7d4698009fec807dab513f7d"} Jul 07 04:56:40.785962 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:40.785959 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:40.804665 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:40.804617 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" podStartSLOduration=2.318703206 podStartE2EDuration="2.804602687s" totalImagesPullingTime="485.899481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:56:40.802718634 +0000 UTC m=+1973.784652488" watchObservedRunningTime="2026-07-07 04:56:40.804602687 +0000 UTC m=+1973.786536538" Jul 07 04:56:46.796247 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:46.796220 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:52.117705 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.117662 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:52.118240 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.117963 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" podUID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" containerName="maas-api" containerID="cri-o://5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a" gracePeriod=30 Jul 07 04:56:52.369826 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.369749 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:52.424024 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.423998 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca\") pod \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " Jul 07 04:56:52.424161 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.424047 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9glb\" (UniqueName: \"kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb\") pod \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " Jul 07 04:56:52.424161 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.424078 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") pod \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\" (UID: \"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962\") " Jul 07 04:56:52.424373 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.424352 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" (UID: "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:56:52.424436 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.424407 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:52.426148 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.426117 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb" (OuterVolumeSpecName: "kube-api-access-c9glb") pod "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" (UID: "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962"). InnerVolumeSpecName "kube-api-access-c9glb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:56:52.426296 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.426262 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" (UID: "c4d97be6-23ec-4a27-b5eb-b0c48a9b2962"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:56:52.524899 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.524874 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9glb\" (UniqueName: \"kubernetes.io/projected/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-kube-api-access-c9glb\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:52.524899 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.524895 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:52.847430 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.847400 2563 generic.go:356] "Generic (PLEG): container finished" podID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" containerID="5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a" exitCode=0 Jul 07 04:56:52.847586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.847443 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" Jul 07 04:56:52.847586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.847480 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" event={"ID":"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962","Type":"ContainerDied","Data":"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a"} Jul 07 04:56:52.847586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.847533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4" event={"ID":"c4d97be6-23ec-4a27-b5eb-b0c48a9b2962","Type":"ContainerDied","Data":"c279d5c7383ff9a15b0443bb0fa4a7fc7d9182efbd5ae59146621b2e214195f6"} Jul 07 04:56:52.847586 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.847558 2563 scope.go:117] "RemoveContainer" containerID="5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a" Jul 07 04:56:52.856557 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.856540 2563 scope.go:117] "RemoveContainer" containerID="5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a" Jul 07 04:56:52.856827 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:52.856807 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a\": container with ID starting with 5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a not found: ID does not exist" containerID="5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a" Jul 07 04:56:52.856893 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.856835 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a"} err="failed to get container status \"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a\": rpc error: code = NotFound desc = could not find container \"5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a\": container with ID starting with 5d267c69f8c129344b4f2da648d242663df5af66131bd9ed5cf9ea46e15ee43a not found: ID does not exist" Jul 07 04:56:52.867539 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.867521 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:52.871881 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:52.871863 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-073254-6f85bf6d87-c2ms4"] Jul 07 04:56:53.528149 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:53.528118 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" path="/var/lib/kubelet/pods/c4d97be6-23ec-4a27-b5eb-b0c48a9b2962/volumes" Jul 07 04:56:58.798550 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:58.798516 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:56:58.798982 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:58.798784 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" podUID="23020b0b-75ef-4b52-9375-d5c36578aa39" containerName="maas-api" containerID="cri-o://1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f" gracePeriod=30 Jul 07 04:56:59.046262 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.046237 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:59.070604 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.070549 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9h2t\" (UniqueName: \"kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t\") pod \"23020b0b-75ef-4b52-9375-d5c36578aa39\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " Jul 07 04:56:59.070604 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.070602 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") pod \"23020b0b-75ef-4b52-9375-d5c36578aa39\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " Jul 07 04:56:59.070799 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.070632 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca\") pod \"23020b0b-75ef-4b52-9375-d5c36578aa39\" (UID: \"23020b0b-75ef-4b52-9375-d5c36578aa39\") " Jul 07 04:56:59.071025 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.070997 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23020b0b-75ef-4b52-9375-d5c36578aa39" (UID: "23020b0b-75ef-4b52-9375-d5c36578aa39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:56:59.072553 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.072530 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23020b0b-75ef-4b52-9375-d5c36578aa39" (UID: "23020b0b-75ef-4b52-9375-d5c36578aa39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:56:59.072674 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.072532 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t" (OuterVolumeSpecName: "kube-api-access-c9h2t") pod "23020b0b-75ef-4b52-9375-d5c36578aa39" (UID: "23020b0b-75ef-4b52-9375-d5c36578aa39"). InnerVolumeSpecName "kube-api-access-c9h2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:56:59.171112 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.171085 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23020b0b-75ef-4b52-9375-d5c36578aa39-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:59.171112 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.171107 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9h2t\" (UniqueName: \"kubernetes.io/projected/23020b0b-75ef-4b52-9375-d5c36578aa39-kube-api-access-c9h2t\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:59.171112 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.171117 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23020b0b-75ef-4b52-9375-d5c36578aa39-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 04:56:59.890759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.890671 2563 generic.go:356] "Generic (PLEG): container finished" podID="23020b0b-75ef-4b52-9375-d5c36578aa39" containerID="1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f" exitCode=0 Jul 07 04:56:59.890759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.890721 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" Jul 07 04:56:59.891287 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.890763 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" event={"ID":"23020b0b-75ef-4b52-9375-d5c36578aa39","Type":"ContainerDied","Data":"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f"} Jul 07 04:56:59.891287 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.890847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m" event={"ID":"23020b0b-75ef-4b52-9375-d5c36578aa39","Type":"ContainerDied","Data":"c1048fcb8918aed1dd1b9cab3e1c960179d73cdd7d4698009fec807dab513f7d"} Jul 07 04:56:59.891287 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.890881 2563 scope.go:117] "RemoveContainer" containerID="1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f" Jul 07 04:56:59.900211 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.900196 2563 scope.go:117] "RemoveContainer" containerID="1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f" Jul 07 04:56:59.900490 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:56:59.900468 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f\": container with ID starting with 1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f not found: ID does not exist" containerID="1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f" Jul 07 04:56:59.900536 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.900501 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f"} err="failed to get container status \"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f\": rpc error: code = NotFound desc = could not find container \"1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f\": container with ID starting with 1658a9e6b3dde821c5ac1c10ac93401fcb59307649935b085f5c5f7e7e2c244f not found: ID does not exist" Jul 07 04:56:59.907636 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.907614 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:56:59.910399 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:56:59.910379 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-59dc16-7c58659fc-x6h2m"] Jul 07 04:57:01.527600 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:01.527561 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23020b0b-75ef-4b52-9375-d5c36578aa39" path="/var/lib/kubelet/pods/23020b0b-75ef-4b52-9375-d5c36578aa39/volumes" Jul 07 04:57:08.144822 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.144740 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145187 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" containerName="maas-api" Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145202 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" containerName="maas-api" Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145213 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23020b0b-75ef-4b52-9375-d5c36578aa39" containerName="maas-api" Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145218 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23020b0b-75ef-4b52-9375-d5c36578aa39" containerName="maas-api" Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145285 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23020b0b-75ef-4b52-9375-d5c36578aa39" containerName="maas-api" Jul 07 04:57:08.145468 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.145293 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d97be6-23ec-4a27-b5eb-b0c48a9b2962" containerName="maas-api" Jul 07 04:57:08.148451 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.148433 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.150922 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.150899 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-41807c-serving-cert\"" Jul 07 04:57:08.160191 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.160168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 04:57:08.243404 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.243374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.243510 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.243411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.243556 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.243510 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gjq\" (UniqueName: \"kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.344100 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.344066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gjq\" (UniqueName: \"kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.344246 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.344111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.344246 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.344131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.344709 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.344688 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.346620 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.346598 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.352447 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.352426 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gjq\" (UniqueName: \"kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq\") pod \"maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.459609 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.459557 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:08.592662 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.592638 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 04:57:08.595023 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:57:08.594989 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381ef025_02d6_4d9e_bd6f_7e29c2eee5f3.slice/crio-0fd2484fbe6cbab5072792a8a4e4e5baed0677a4c0616439ee200e9cd8a8d872 WatchSource:0}: Error finding container 0fd2484fbe6cbab5072792a8a4e4e5baed0677a4c0616439ee200e9cd8a8d872: Status 404 returned error can't find the container with id 0fd2484fbe6cbab5072792a8a4e4e5baed0677a4c0616439ee200e9cd8a8d872 Jul 07 04:57:08.937355 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:08.937319 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" event={"ID":"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3","Type":"ContainerStarted","Data":"0fd2484fbe6cbab5072792a8a4e4e5baed0677a4c0616439ee200e9cd8a8d872"} Jul 07 04:57:09.944135 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:09.944100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" event={"ID":"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3","Type":"ContainerStarted","Data":"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0"} Jul 07 04:57:09.944498 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:09.944216 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:09.964806 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:09.964747 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" podStartSLOduration=1.636461398 podStartE2EDuration="1.964731604s" totalImagesPullingTime="328.270206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:57:09.963459898 +0000 UTC m=+2002.945393750" watchObservedRunningTime="2026-07-07 04:57:09.964731604 +0000 UTC m=+2002.946665457" Jul 07 04:57:15.955243 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:15.955207 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 04:57:16.202515 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.202469 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 04:57:16.206786 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.206707 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.209223 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.209189 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-5a77f6-serving-cert\"" Jul 07 04:57:16.215182 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.215158 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 04:57:16.310422 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.310391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.310579 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.310434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.310579 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.310476 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dw5\" (UniqueName: \"kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.411632 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.411599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.411759 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.411641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.411819 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.411799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dw5\" (UniqueName: \"kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.411890 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:57:16.411803 2563 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-5a77f6-serving-cert: secret "maas-api-e2e-shared-b-5a77f6-serving-cert" not found Jul 07 04:57:16.411975 ip-10-0-142-140 kubenswrapper[2563]: E0707 04:57:16.411964 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls podName:3f6ffcac-ad61-4de8-b503-42eb8be363b9 nodeName:}" failed. No retries permitted until 2026-07-07 04:57:16.911940942 +0000 UTC m=+2009.893874786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls") pod "maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" (UID: "3f6ffcac-ad61-4de8-b503-42eb8be363b9") : secret "maas-api-e2e-shared-b-5a77f6-serving-cert" not found Jul 07 04:57:16.412362 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.412345 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.420231 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.420208 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dw5\" (UniqueName: \"kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.915799 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.915752 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:16.918297 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:16.918273 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:17.121457 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:17.121424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:17.257444 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:17.257418 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 04:57:17.259869 ip-10-0-142-140 kubenswrapper[2563]: W0707 04:57:17.259827 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6ffcac_ad61_4de8_b503_42eb8be363b9.slice/crio-b48aa0ce1e06b0a13bfcd502d7ee316936a036f27fb1e4b981b5401bf30ad0df WatchSource:0}: Error finding container b48aa0ce1e06b0a13bfcd502d7ee316936a036f27fb1e4b981b5401bf30ad0df: Status 404 returned error can't find the container with id b48aa0ce1e06b0a13bfcd502d7ee316936a036f27fb1e4b981b5401bf30ad0df Jul 07 04:57:17.989640 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:17.989607 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" event={"ID":"3f6ffcac-ad61-4de8-b503-42eb8be363b9","Type":"ContainerStarted","Data":"b48aa0ce1e06b0a13bfcd502d7ee316936a036f27fb1e4b981b5401bf30ad0df"} Jul 07 04:57:18.995311 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:18.995273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" event={"ID":"3f6ffcac-ad61-4de8-b503-42eb8be363b9","Type":"ContainerStarted","Data":"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f"} Jul 07 04:57:18.995757 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:18.995345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:57:19.015444 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:19.015393 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" podStartSLOduration=2.154328677 podStartE2EDuration="3.015378825s" totalImagesPullingTime="861.050148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:57:19.014486522 +0000 UTC m=+2011.996420398" watchObservedRunningTime="2026-07-07 04:57:19.015378825 +0000 UTC m=+2011.997312677" Jul 07 04:57:25.006313 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:57:25.006285 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 04:58:47.612280 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:58:47.612243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:58:47.614717 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:58:47.614347 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 04:58:47.687100 ip-10-0-142-140 kubenswrapper[2563]: I0707 04:58:47.687076 2563 scope.go:117] "RemoveContainer" containerID="215408116d9ae8e82fc403d50ec18297058468f87f3c2358b8fee39659a64261" Jul 07 05:00:00.141237 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.141201 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723340-zsh56"] Jul 07 05:00:00.144592 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.144572 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:00.155631 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.155603 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723340-zsh56"] Jul 07 05:00:00.312940 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.312894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5xd\" (UniqueName: \"kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd\") pod \"maas-api-key-cleanup-29723340-zsh56\" (UID: \"e2895f04-7d26-471b-913b-43b5bf064d3f\") " pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:00.413901 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.413816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5xd\" (UniqueName: \"kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd\") pod \"maas-api-key-cleanup-29723340-zsh56\" (UID: \"e2895f04-7d26-471b-913b-43b5bf064d3f\") " pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:00.421844 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.421809 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5xd\" (UniqueName: \"kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd\") pod \"maas-api-key-cleanup-29723340-zsh56\" (UID: \"e2895f04-7d26-471b-913b-43b5bf064d3f\") " pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:00.456542 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.456518 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:00.590948 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.590915 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723340-zsh56"] Jul 07 05:00:00.593881 ip-10-0-142-140 kubenswrapper[2563]: W0707 05:00:00.593841 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2895f04_7d26_471b_913b_43b5bf064d3f.slice/crio-2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a WatchSource:0}: Error finding container 2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a: Status 404 returned error can't find the container with id 2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a Jul 07 05:00:00.775487 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:00.775457 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" event={"ID":"e2895f04-7d26-471b-913b-43b5bf064d3f","Type":"ContainerStarted","Data":"2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a"} Jul 07 05:00:01.782437 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:01.782402 2563 generic.go:356] "Generic (PLEG): container finished" podID="e2895f04-7d26-471b-913b-43b5bf064d3f" containerID="00512022a1965bf0cfc05f539275061a3825c68409438dc49e202da417de2c6e" exitCode=0 Jul 07 05:00:01.782891 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:01.782444 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" event={"ID":"e2895f04-7d26-471b-913b-43b5bf064d3f","Type":"ContainerDied","Data":"00512022a1965bf0cfc05f539275061a3825c68409438dc49e202da417de2c6e"} Jul 07 05:00:02.930644 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:02.930619 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:03.035074 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.035036 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n5xd\" (UniqueName: \"kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd\") pod \"e2895f04-7d26-471b-913b-43b5bf064d3f\" (UID: \"e2895f04-7d26-471b-913b-43b5bf064d3f\") " Jul 07 05:00:03.037105 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.037073 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd" (OuterVolumeSpecName: "kube-api-access-6n5xd") pod "e2895f04-7d26-471b-913b-43b5bf064d3f" (UID: "e2895f04-7d26-471b-913b-43b5bf064d3f"). InnerVolumeSpecName "kube-api-access-6n5xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:00:03.136508 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.136467 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5xd\" (UniqueName: \"kubernetes.io/projected/e2895f04-7d26-471b-913b-43b5bf064d3f-kube-api-access-6n5xd\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:00:03.793171 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.793083 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" Jul 07 05:00:03.793171 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.793118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723340-zsh56" event={"ID":"e2895f04-7d26-471b-913b-43b5bf064d3f","Type":"ContainerDied","Data":"2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a"} Jul 07 05:00:03.793356 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:00:03.793171 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f4909871e9f5c54ed43cdd6e61d7fc9f61e4694ab61baf76a30c5edf0806f6a" Jul 07 05:02:39.078606 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.078567 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 05:02:39.079449 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.079393 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" podUID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" containerName="maas-api" containerID="cri-o://3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0" gracePeriod=30 Jul 07 05:02:39.332684 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.332627 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 05:02:39.370271 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.370240 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2gjq\" (UniqueName: \"kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq\") pod \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " Jul 07 05:02:39.370391 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.370322 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls\") pod \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " Jul 07 05:02:39.370391 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.370355 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca\") pod \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\" (UID: \"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3\") " Jul 07 05:02:39.370759 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.370734 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" (UID: "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 05:02:39.372323 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.372289 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq" (OuterVolumeSpecName: "kube-api-access-c2gjq") pod "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" (UID: "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3"). InnerVolumeSpecName "kube-api-access-c2gjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:02:39.372323 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.372296 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" (UID: "381ef025-02d6-4d9e-bd6f-7e29c2eee5f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:02:39.471122 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.471095 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:02:39.471122 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.471121 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:02:39.471334 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.471132 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gjq\" (UniqueName: \"kubernetes.io/projected/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3-kube-api-access-c2gjq\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:02:39.552604 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.552575 2563 generic.go:356] "Generic (PLEG): container finished" podID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" containerID="3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0" exitCode=0 Jul 07 05:02:39.552720 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.552614 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" Jul 07 05:02:39.552720 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.552648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" event={"ID":"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3","Type":"ContainerDied","Data":"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0"} Jul 07 05:02:39.552720 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.552692 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g" event={"ID":"381ef025-02d6-4d9e-bd6f-7e29c2eee5f3","Type":"ContainerDied","Data":"0fd2484fbe6cbab5072792a8a4e4e5baed0677a4c0616439ee200e9cd8a8d872"} Jul 07 05:02:39.552720 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.552715 2563 scope.go:117] "RemoveContainer" containerID="3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0" Jul 07 05:02:39.561354 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.561335 2563 scope.go:117] "RemoveContainer" containerID="3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0" Jul 07 05:02:39.561618 ip-10-0-142-140 kubenswrapper[2563]: E0707 05:02:39.561599 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0\": container with ID starting with 3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0 not found: ID does not exist" containerID="3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0" Jul 07 05:02:39.561680 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.561626 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0"} err="failed to get container status \"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0\": rpc error: code = NotFound desc = could not find container \"3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0\": container with ID starting with 3d8b19d5cc106583fb3fec229439032e981e31e711dcd5046e02022e9fb97bb0 not found: ID does not exist" Jul 07 05:02:39.569498 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.569475 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 05:02:39.572801 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:39.572755 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-41807c-57455d54bd-xqb5g"] Jul 07 05:02:41.528668 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:02:41.528632 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" path="/var/lib/kubelet/pods/381ef025-02d6-4d9e-bd6f-7e29c2eee5f3/volumes" Jul 07 05:03:21.182323 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.182286 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 05:03:21.182934 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.182588 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" podUID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" containerName="maas-api" containerID="cri-o://3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f" gracePeriod=30 Jul 07 05:03:21.435971 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.435913 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 05:03:21.499589 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.499559 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5dw5\" (UniqueName: \"kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5\") pod \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " Jul 07 05:03:21.499710 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.499625 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca\") pod \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " Jul 07 05:03:21.499710 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.499691 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") pod \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\" (UID: \"3f6ffcac-ad61-4de8-b503-42eb8be363b9\") " Jul 07 05:03:21.500025 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.499999 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f6ffcac-ad61-4de8-b503-42eb8be363b9" (UID: "3f6ffcac-ad61-4de8-b503-42eb8be363b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 05:03:21.501657 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.501634 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5" (OuterVolumeSpecName: "kube-api-access-k5dw5") pod "3f6ffcac-ad61-4de8-b503-42eb8be363b9" (UID: "3f6ffcac-ad61-4de8-b503-42eb8be363b9"). InnerVolumeSpecName "kube-api-access-k5dw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:03:21.501657 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.501648 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f6ffcac-ad61-4de8-b503-42eb8be363b9" (UID: "3f6ffcac-ad61-4de8-b503-42eb8be363b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:03:21.600628 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.600600 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ffcac-ad61-4de8-b503-42eb8be363b9-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:03:21.600628 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.600627 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6ffcac-ad61-4de8-b503-42eb8be363b9-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:03:21.600758 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.600642 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5dw5\" (UniqueName: \"kubernetes.io/projected/3f6ffcac-ad61-4de8-b503-42eb8be363b9-kube-api-access-k5dw5\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\"" Jul 07 05:03:21.754924 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.754897 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" containerID="3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f" exitCode=0 Jul 07 05:03:21.755049 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.754942 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" Jul 07 05:03:21.755049 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.754956 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" event={"ID":"3f6ffcac-ad61-4de8-b503-42eb8be363b9","Type":"ContainerDied","Data":"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f"} Jul 07 05:03:21.755049 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.755017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk" event={"ID":"3f6ffcac-ad61-4de8-b503-42eb8be363b9","Type":"ContainerDied","Data":"b48aa0ce1e06b0a13bfcd502d7ee316936a036f27fb1e4b981b5401bf30ad0df"} Jul 07 05:03:21.755213 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.755050 2563 scope.go:117] "RemoveContainer" containerID="3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f" Jul 07 05:03:21.764217 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.764196 2563 scope.go:117] "RemoveContainer" containerID="3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f" Jul 07 05:03:21.764479 ip-10-0-142-140 kubenswrapper[2563]: E0707 05:03:21.764457 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f\": container with ID starting with 3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f not found: ID does not exist" containerID="3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f" Jul 07 05:03:21.764523 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.764488 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f"} err="failed to get container status \"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f\": rpc error: code = NotFound desc = could not find container \"3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f\": container with ID starting with 3b52e2d614901155daee42b0ca541288b1759e3685ce436fd90b243436db8b8f not found: ID does not exist" Jul 07 05:03:21.770605 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.770578 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 05:03:21.780478 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:21.780458 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5a77f6-7649f69f86-mgdkk"] Jul 07 05:03:23.528638 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:23.528608 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" path="/var/lib/kubelet/pods/3f6ffcac-ad61-4de8-b503-42eb8be363b9/volumes" Jul 07 05:03:33.054720 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:33.054685 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-54gzb_f9f59874-d250-49e5-9dab-d2c9d93ae8c6/manager/0.log" Jul 07 05:03:33.181192 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:33.181165 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69668d87b5-7sgxk_8651ce06-ffbc-4759-a217-49972bf1ffe2/maas-api/0.log" Jul 07 05:03:33.290016 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:33.289992 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723325-fthkq_3c1a6225-f8b6-43c8-ac3e-da0d582b3faa/cleanup/0.log" Jul 07 05:03:33.407435 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:33.407376 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723340-zsh56_e2895f04-7d26-471b-913b-43b5bf064d3f/cleanup/0.log" Jul 07 05:03:33.534840 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:33.534804 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65fd5f6bc4-ll7cz_f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4/manager/0.log" Jul 07 05:03:34.013683 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:34.013659 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-td7dh_2933a371-7078-4e24-92c2-c252edd93139/copy-manifests/0.log" Jul 07 05:03:34.021586 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:34.021561 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-td7dh_2933a371-7078-4e24-92c2-c252edd93139/manager/0.log" Jul 07 05:03:35.669402 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:35.669370 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ksdkk_b2d6b471-8323-4d43-95b8-8f2d3492ca44/manager/0.log" Jul 07 05:03:36.396211 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:36.396181 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vblh7_f4413624-b0c1-4510-8c73-17aa22855626/manager/0.log" Jul 07 05:03:36.870847 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:36.870817 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z8xpb_79040cef-bf94-40c4-bbca-cd0893cef2cd/discovery/0.log" Jul 07 05:03:36.982870 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:36.982845 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-5k9mm_103c4428-f60a-4302-81a4-62991fc61a55/kube-auth-proxy/0.log" Jul 07 05:03:37.569251 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:37.569222 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bbcb5b45-x2k47_15545f1f-bb57-4d18-8c65-cdf59a7a5dc9/router/0.log" Jul 07 05:03:37.914967 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:37.914894 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6b7d2_cbb0837e-669d-4e4f-8827-65d671f97e2c/main/0.log" Jul 07 05:03:38.245045 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:38.245005 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbffbn9_e56a4cc3-1d34-4256-a86b-c49f9f6f07bc/main/0.log" Jul 07 05:03:38.356216 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:38.356187 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gcsxs_7972eb63-782d-4ae5-86b5-0a5e6564c695/main/0.log" Jul 07 05:03:46.132570 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:46.132536 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2tvqx_8fffa7d6-6a77-4a16-97b1-8c0ee41b1cca/global-pull-secret-syncer/0.log" Jul 07 05:03:46.302316 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:46.302289 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mvhjd_71128bfb-74e3-45f7-a7c1-eca531c3cad8/konnectivity-agent/0.log" Jul 07 05:03:46.404155 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:46.404076 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-140.ec2.internal_44f23d15227f2b810bc3ab580d4a5aaa/haproxy/0.log" Jul 07 05:03:47.640052 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:47.639949 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 05:03:47.649877 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:47.649834 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 05:03:51.737843 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:51.737797 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ksdkk_b2d6b471-8323-4d43-95b8-8f2d3492ca44/manager/0.log" Jul 07 05:03:52.088187 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:52.088090 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vblh7_f4413624-b0c1-4510-8c73-17aa22855626/manager/0.log" Jul 07 05:03:53.933276 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:53.933177 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-f8zbb_ecd39bb2-bc3b-481f-97fd-96ee13e13f27/cluster-monitoring-operator/0.log" Jul 07 05:03:54.255724 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:54.255694 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv2f_16682724-bbc5-423f-aff3-79b1865d19bf/node-exporter/0.log" Jul 07 05:03:54.275432 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:54.275413 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv2f_16682724-bbc5-423f-aff3-79b1865d19bf/kube-rbac-proxy/0.log" Jul 07 05:03:54.296640 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:54.296619 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv2f_16682724-bbc5-423f-aff3-79b1865d19bf/init-textfile/0.log" Jul 07 05:03:55.945829 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:55.945801 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-npwxk_092433f8-30e4-4c3c-919c-901d32b4a390/networking-console-plugin/0.log" Jul 07 05:03:56.523189 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.523161 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/1.log" Jul 07 05:03:56.529719 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.529689 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw"] Jul 07 05:03:56.530229 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530200 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2895f04-7d26-471b-913b-43b5bf064d3f" containerName="cleanup" Jul 07 05:03:56.530229 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530224 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2895f04-7d26-471b-913b-43b5bf064d3f" containerName="cleanup" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530262 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" containerName="maas-api" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530270 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" containerName="maas-api" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530284 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" containerName="maas-api" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530293 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" containerName="maas-api" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530375 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="381ef025-02d6-4d9e-bd6f-7e29c2eee5f3" containerName="maas-api" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530389 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2895f04-7d26-471b-913b-43b5bf064d3f" containerName="cleanup" Jul 07 05:03:56.530416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.530399 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f6ffcac-ad61-4de8-b503-42eb8be363b9" containerName="maas-api" Jul 07 05:03:56.533233 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.533213 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cp2fz_a92b02e2-395b-44df-89ec-8d335b6e26fc/console-operator/2.log" Jul 07 05:03:56.533731 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.533714 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.535915 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.535893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh4c7\"/\"kube-root-ca.crt\"" Jul 07 05:03:56.536678 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.536657 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jh4c7\"/\"default-dockercfg-hblzf\"" Jul 07 05:03:56.536784 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.536675 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh4c7\"/\"openshift-service-ca.crt\"" Jul 07 05:03:56.543255 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.543222 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw"] Jul 07 05:03:56.577314 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.577290 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-proc\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.577416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.577318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-lib-modules\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.577416 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.577385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-sys\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.577576 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.577560 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4sjk\" (UniqueName: \"kubernetes.io/projected/9e5310af-285e-4023-848b-d4a6c26cdbf9-kube-api-access-w4sjk\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.577644 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.577598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-podres\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678526 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4sjk\" (UniqueName: \"kubernetes.io/projected/9e5310af-285e-4023-848b-d4a6c26cdbf9-kube-api-access-w4sjk\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678618 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-podres\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678618 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-proc\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678724 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-lib-modules\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678724 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-sys\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678724 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678709 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-proc\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678887 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678749 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-lib-modules\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678887 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678760 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-podres\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.678887 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.678810 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5310af-285e-4023-848b-d4a6c26cdbf9-sys\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.685943 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.685926 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4sjk\" (UniqueName: \"kubernetes.io/projected/9e5310af-285e-4023-848b-d4a6c26cdbf9-kube-api-access-w4sjk\") pod \"perf-node-gather-daemonset-m49hw\" (UID: \"9e5310af-285e-4023-848b-d4a6c26cdbf9\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.844551 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.844493 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:56.973046 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:56.973015 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw"] Jul 07 05:03:56.976423 ip-10-0-142-140 kubenswrapper[2563]: W0707 05:03:56.976397 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e5310af_285e_4023_848b_d4a6c26cdbf9.slice/crio-4c882e8a42b271d34c6d15d25ce5d6f00508ef61cf5f6102e483165c9809a672 WatchSource:0}: Error finding container 4c882e8a42b271d34c6d15d25ce5d6f00508ef61cf5f6102e483165c9809a672: Status 404 returned error can't find the container with id 4c882e8a42b271d34c6d15d25ce5d6f00508ef61cf5f6102e483165c9809a672 Jul 07 05:03:57.620215 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:57.620188 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-s9hlv_cab9ef11-19ee-4259-9fb0-f68c45be6f75/volume-data-source-validator/0.log" Jul 07 05:03:57.932604 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:57.932521 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" event={"ID":"9e5310af-285e-4023-848b-d4a6c26cdbf9","Type":"ContainerStarted","Data":"dd85aa1f986ddd5b653001c63b137f4c43fb7a31e896ee257668c708dfd5045f"} Jul 07 05:03:57.932604 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:57.932593 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" event={"ID":"9e5310af-285e-4023-848b-d4a6c26cdbf9","Type":"ContainerStarted","Data":"4c882e8a42b271d34c6d15d25ce5d6f00508ef61cf5f6102e483165c9809a672"} Jul 07 05:03:57.932853 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:57.932674 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:03:57.950587 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:57.950537 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" podStartSLOduration=1.950524729 podStartE2EDuration="1.950524729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 05:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:03:57.948552503 +0000 UTC m=+2410.930486356" watchObservedRunningTime="2026-07-07 05:03:57.950524729 +0000 UTC m=+2410.932458580" Jul 07 05:03:58.621703 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:58.621676 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72lzq_1f3fc374-e44f-4652-9308-c1b1036a56cc/dns/0.log" Jul 07 05:03:58.662729 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:58.662706 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72lzq_1f3fc374-e44f-4652-9308-c1b1036a56cc/kube-rbac-proxy/0.log" Jul 07 05:03:58.892589 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:58.892519 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvmr2_e58e6ac7-8ee6-4790-8625-54d10183203e/dns-node-resolver/0.log" Jul 07 05:03:59.539990 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:03:59.539950 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c8tf5_42292ea1-1500-4541-9321-06324275ce72/node-ca/0.log" Jul 07 05:04:00.676656 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:00.676624 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z8xpb_79040cef-bf94-40c4-bbca-cd0893cef2cd/discovery/0.log" Jul 07 05:04:00.696164 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:00.696145 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-5k9mm_103c4428-f60a-4302-81a4-62991fc61a55/kube-auth-proxy/0.log" Jul 07 05:04:00.876222 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:00.876195 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bbcb5b45-x2k47_15545f1f-bb57-4d18-8c65-cdf59a7a5dc9/router/0.log" Jul 07 05:04:01.587438 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:01.587412 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkxhb_8aac71c4-980e-4b85-a65f-e40c93af620c/serve-healthcheck-canary/0.log" Jul 07 05:04:02.037442 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:02.037413 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-pswrc_d9049628-5100-4314-abaa-bb11f6760a4e/insights-operator/0.log" Jul 07 05:04:02.189394 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:02.189353 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tmlwm_49ba2da0-f024-44f3-a481-4a9ce844b390/kube-rbac-proxy/0.log" Jul 07 05:04:02.211698 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:02.211673 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tmlwm_49ba2da0-f024-44f3-a481-4a9ce844b390/exporter/0.log" Jul 07 05:04:02.231729 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:02.231706 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tmlwm_49ba2da0-f024-44f3-a481-4a9ce844b390/extractor/0.log" Jul 07 05:04:03.947161 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:03.947134 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-m49hw" Jul 07 05:04:04.599343 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:04.599319 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-54gzb_f9f59874-d250-49e5-9dab-d2c9d93ae8c6/manager/0.log" Jul 07 05:04:04.646228 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:04.646192 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69668d87b5-7sgxk_8651ce06-ffbc-4759-a217-49972bf1ffe2/maas-api/0.log" Jul 07 05:04:04.665745 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:04.665722 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723325-fthkq_3c1a6225-f8b6-43c8-ac3e-da0d582b3faa/cleanup/0.log" Jul 07 05:04:04.689856 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:04.689837 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723340-zsh56_e2895f04-7d26-471b-913b-43b5bf064d3f/cleanup/0.log" Jul 07 05:04:04.855215 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:04.855140 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65fd5f6bc4-ll7cz_f1a1af7b-1109-4e1b-9ae4-b28d7421bcd4/manager/0.log" Jul 07 05:04:05.042808 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:05.042778 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-td7dh_2933a371-7078-4e24-92c2-c252edd93139/manager/0.log" Jul 07 05:04:05.062867 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:05.062848 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-td7dh_2933a371-7078-4e24-92c2-c252edd93139/copy-manifests/0.log" Jul 07 05:04:07.164081 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:07.164051 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z8xpb_79040cef-bf94-40c4-bbca-cd0893cef2cd/discovery/0.log" Jul 07 05:04:07.173457 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:07.173430 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-5k9mm_103c4428-f60a-4302-81a4-62991fc61a55/kube-auth-proxy/0.log" Jul 07 05:04:07.305124 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:07.305098 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bbcb5b45-x2k47_15545f1f-bb57-4d18-8c65-cdf59a7a5dc9/router/0.log" Jul 07 05:04:08.693847 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:08.693813 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-z8xpb_79040cef-bf94-40c4-bbca-cd0893cef2cd/discovery/0.log" Jul 07 05:04:08.713936 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:08.713914 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-5k9mm_103c4428-f60a-4302-81a4-62991fc61a55/kube-auth-proxy/0.log" Jul 07 05:04:08.861415 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:08.861391 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bbcb5b45-x2k47_15545f1f-bb57-4d18-8c65-cdf59a7a5dc9/router/0.log" Jul 07 05:04:11.754013 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:11.753929 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-nrmlc_bb4bc44d-0b6b-44db-8476-f04f39c5c9a4/migrator/0.log" Jul 07 05:04:11.773402 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:11.773380 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-nrmlc_bb4bc44d-0b6b-44db-8476-f04f39c5c9a4/graceful-termination/0.log" Jul 07 05:04:12.115653 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:12.115576 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-wnvlz_2c61aea7-54b3-4024-8f7d-5b1a0d806617/kube-storage-version-migrator-operator/1.log" Jul 07 05:04:12.117270 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:12.117239 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-wnvlz_2c61aea7-54b3-4024-8f7d-5b1a0d806617/kube-storage-version-migrator-operator/0.log" Jul 07 05:04:13.388103 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.388072 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/kube-multus-additional-cni-plugins/0.log" Jul 07 05:04:13.407738 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.407712 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/egress-router-binary-copy/0.log" Jul 07 05:04:13.427058 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.427040 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/cni-plugins/0.log" Jul 07 05:04:13.446210 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.446193 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/bond-cni-plugin/0.log" Jul 07 05:04:13.467722 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.467706 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/routeoverride-cni/0.log" Jul 07 05:04:13.488539 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.488519 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/whereabouts-cni-bincopy/0.log" Jul 07 05:04:13.508651 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.508636 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z8fdh_d8436039-fc85-4333-88f1-d8ab9671e473/whereabouts-cni/0.log" Jul 07 05:04:13.548202 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.548158 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l26zw_c6af0dfb-4a51-4173-8611-6b091dd6751c/kube-multus/0.log" Jul 07 05:04:13.629904 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.629866 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7mhwl_137a0240-63ec-41ac-910d-aafdbcfc59f0/network-metrics-daemon/0.log" Jul 07 05:04:13.645230 ip-10-0-142-140 kubenswrapper[2563]: I0707 05:04:13.645169 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7mhwl_137a0240-63ec-41ac-910d-aafdbcfc59f0/kube-rbac-proxy/0.log"