Jul 01 21:09:39.396751 ip-10-0-140-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 01 21:09:39.396763 ip-10-0-140-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 01 21:09:39.396770 ip-10-0-140-112 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 01 21:09:39.396982 ip-10-0-140-112 systemd[1]: Failed to start Kubernetes Kubelet. Jul 01 21:09:49.522093 ip-10-0-140-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 01 21:09:49.522113 ip-10-0-140-112 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 03b75ac083e1496694d178a5e87ccec0 -- Jul 01 21:11:39.364658 ip-10-0-140-112 systemd[1]: Starting Kubernetes Kubelet... Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.284362 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290354 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290368 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290373 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290376 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290379 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290382 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290385 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.358137 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290388 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290391 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290394 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290396 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290399 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290401 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290404 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290407 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290410 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290412 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290415 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290417 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290420 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290422 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290425 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290428 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290430 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290433 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290436 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290439 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.359143 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290441 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290444 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290461 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290464 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290467 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290469 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290474 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290478 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290481 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290484 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290487 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290490 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290492 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290496 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290498 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290501 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290504 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290506 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290508 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290511 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.359752 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290514 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290517 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290519 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290522 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290524 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290530 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290533 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290537 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290539 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290542 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290546 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290549 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290551 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290554 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290557 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290561 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290563 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290566 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290569 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.360361 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290572 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290574 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290577 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290580 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290583 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290585 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290588 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290590 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290593 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290596 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290598 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290601 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290603 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290606 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290615 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290617 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290620 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290623 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290626 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290628 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.360972 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290631 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290633 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290636 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290639 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290642 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290645 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290648 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290652 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290655 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290658 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290660 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290663 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.290665 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291479 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291486 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291498 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291502 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291505 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291508 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.361502 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291510 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291513 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291516 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291519 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291521 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291523 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291526 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291528 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291538 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291541 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291544 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291547 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291549 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291552 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291554 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291557 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291560 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291562 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291565 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291567 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.362014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291570 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291572 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291575 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291578 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291580 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291583 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291585 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291588 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291590 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291592 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291595 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291597 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291600 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291602 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291604 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291607 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291613 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291615 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291618 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291620 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.362566 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291623 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291631 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291634 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291644 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291647 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291649 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291652 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291655 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291657 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291660 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291663 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291665 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291668 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291671 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291673 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291676 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291678 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291682 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291686 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.363123 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291689 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291692 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291694 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291696 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291699 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291702 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291705 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291707 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291710 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291714 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291717 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291719 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291721 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291724 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291726 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291737 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291740 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291743 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291746 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291749 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.363954 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291751 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291755 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291758 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291761 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291764 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291768 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291771 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291774 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291776 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291779 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291781 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291784 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291787 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.291789 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292215 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292220 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292226 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292230 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292233 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.364553 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292236 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292238 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292243 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292246 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292248 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292251 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292254 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292256 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292260 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292264 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292267 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292270 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292273 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292276 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292283 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292285 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292289 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292291 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292294 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292297 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.469935 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292299 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.380238 ip-10-0-140-112 systemd[1]: Started Kubernetes Kubelet. Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292302 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292305 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292308 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292310 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292313 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292316 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292319 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292321 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292324 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292327 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292329 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292333 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292335 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292339 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292342 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292345 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292347 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292350 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292352 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.486756 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292354 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292358 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292360 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292363 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292365 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292367 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292370 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292372 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292375 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292377 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292380 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292382 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292385 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292388 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292391 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292393 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292396 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292398 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292401 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.487663 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292404 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292407 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292410 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292413 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292416 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292418 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292420 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292425 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292428 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292430 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292433 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292435 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292437 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292441 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292443 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292447 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292449 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292452 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292454 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292457 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.488339 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292459 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292462 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292465 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292467 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292470 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292474 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292477 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292479 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292482 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292485 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292487 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292489 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292492 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292494 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.292497 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292578 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292586 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292593 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292597 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292604 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292607 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 01 21:11:41.489038 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292612 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292616 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292619 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292622 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292626 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292630 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292633 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292641 2577 flags.go:64] FLAG: --cgroup-root="" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292645 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292648 2577 flags.go:64] FLAG: --client-ca-file="" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292651 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292653 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292659 2577 flags.go:64] FLAG: --cluster-domain="" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292662 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292665 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292668 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292671 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292676 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292679 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292683 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292686 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292689 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292692 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292695 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 01 21:11:41.489724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292698 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292701 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292706 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292709 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292712 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292714 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292719 2577 flags.go:64] FLAG: --enable-server="true" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292722 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292726 2577 flags.go:64] FLAG: --event-burst="100" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292729 2577 flags.go:64] FLAG: --event-qps="50" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292732 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292735 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292738 2577 flags.go:64] FLAG: --eviction-hard="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292748 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292750 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292755 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292759 2577 flags.go:64] FLAG: --eviction-soft="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292762 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292765 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292768 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292770 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292773 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292776 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292779 2577 flags.go:64] FLAG: --feature-gates="" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292783 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 01 21:11:41.490653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292786 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292788 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292792 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292795 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292798 2577 flags.go:64] FLAG: --help="false" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292800 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292803 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292806 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292809 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292813 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292816 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292819 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292821 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292825 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292828 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292831 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292834 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292837 2577 flags.go:64] FLAG: --kube-reserved="" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292839 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292842 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292845 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292848 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292853 2577 flags.go:64] FLAG: --lock-file="" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292855 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 01 21:11:41.491623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292859 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292862 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292867 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292869 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292872 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292875 2577 flags.go:64] FLAG: --logging-format="text" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292878 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292881 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292883 2577 flags.go:64] FLAG: --manifest-url="" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292886 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292891 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292894 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292898 2577 flags.go:64] FLAG: --max-pods="110" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292900 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292903 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292906 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292909 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292912 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292914 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292917 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292925 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292929 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292932 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 01 21:11:41.492389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292934 2577 flags.go:64] FLAG: --pod-cidr="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292937 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292943 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292947 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292950 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292953 2577 flags.go:64] FLAG: --port="10250" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292956 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292959 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ad1d116764d7eac0" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292963 2577 flags.go:64] FLAG: --qos-reserved="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292966 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292969 2577 flags.go:64] FLAG: --register-node="true" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292972 2577 flags.go:64] FLAG: --register-with-taints="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292976 2577 flags.go:64] FLAG: --registry-burst="10" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292979 2577 flags.go:64] FLAG: --registry-qps="5" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292981 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292984 2577 flags.go:64] FLAG: --reserved-memory="" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292987 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292990 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292993 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292996 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.292999 2577 flags.go:64] FLAG: --runonce="false" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293002 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293004 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293007 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293010 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 01 21:11:41.493301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293013 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293015 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293018 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293022 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293024 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293028 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293031 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293034 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293037 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293039 2577 flags.go:64] FLAG: --system-cgroups="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293042 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293047 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293050 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293053 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293057 2577 flags.go:64] FLAG: --tls-min-version="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293062 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293064 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293067 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293071 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293074 2577 flags.go:64] FLAG: --v="2" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293078 2577 flags.go:64] FLAG: --version="false" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293082 2577 flags.go:64] FLAG: --vmodule="" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293085 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.293089 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293221 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.494031 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293225 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293228 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293231 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293234 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293237 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293240 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293242 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293245 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293247 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293250 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293253 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293255 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293259 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293262 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293265 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293267 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293270 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293273 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293275 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293278 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.494813 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293280 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293283 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293288 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293291 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293294 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293296 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293299 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293302 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293304 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293307 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293311 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293315 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293319 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293322 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293325 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293328 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293331 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293334 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293337 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.495467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293340 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293342 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293344 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293347 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293350 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293354 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293356 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293359 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293361 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293364 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293366 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293368 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293371 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293373 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293376 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293380 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293383 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293386 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293389 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293391 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.496448 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293394 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293397 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293399 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293402 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293404 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293408 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293411 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293414 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293417 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293420 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293422 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293425 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293427 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293430 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293432 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293434 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293437 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293441 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293444 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.497080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293446 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293449 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293451 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293454 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293456 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293459 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293461 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293464 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293468 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293470 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293473 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293475 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293478 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293480 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293483 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293485 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293488 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293490 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293493 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.497703 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.293495 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.296928 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.303552 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.303569 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.303586 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.303592 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303668 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303674 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303677 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303680 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303683 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303686 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303689 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303695 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303699 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.498328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303702 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303705 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303707 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303710 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303713 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303715 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303717 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303720 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303723 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303725 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303728 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303731 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303734 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303737 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303740 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303742 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303745 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303747 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303750 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303752 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.498937 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303755 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303758 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303761 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303763 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303766 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303768 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303771 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303773 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303776 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303779 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303783 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303785 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303788 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303791 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303793 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303796 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303798 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303801 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303803 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303806 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.499556 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303808 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303811 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303814 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303816 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303819 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303822 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303825 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303828 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303830 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303833 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303835 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303838 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303840 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303843 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303845 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303848 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303850 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303853 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303855 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303858 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.500466 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303860 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303863 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303868 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303870 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303874 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303878 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303880 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303883 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303885 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303888 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303892 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303895 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303899 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303902 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303905 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303908 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303910 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303913 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303921 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.503141 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303924 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303926 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303929 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303932 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303934 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303937 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303939 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303942 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303944 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303947 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.303949 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.303954 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304090 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304095 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304098 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 01 21:11:41.503760 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304101 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304105 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304110 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304114 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304117 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304120 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304123 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304126 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304129 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304131 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304134 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304137 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304139 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304142 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304145 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304147 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304149 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304153 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304156 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 01 21:11:41.504304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304159 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304161 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304164 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304166 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304169 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304184 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304187 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304190 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304192 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304195 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304198 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304200 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304203 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304205 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304210 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304214 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304216 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304219 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304221 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304224 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 01 21:11:41.504808 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304227 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304229 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304232 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304234 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304237 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304239 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304242 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304244 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304248 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304251 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304255 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304258 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304261 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304264 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304267 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304269 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304272 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304274 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304277 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304279 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 01 21:11:41.505542 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304282 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304284 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304287 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304289 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304291 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304294 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304298 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304301 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304304 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304307 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304309 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304312 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304314 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304317 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304320 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304322 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304325 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304327 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304330 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304333 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 01 21:11:41.506081 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304335 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304338 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304341 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304343 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304346 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304349 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304351 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304354 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304356 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304359 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304362 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304364 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304367 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304369 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304373 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304377 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 01 21:11:41.639902 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:41.304379 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.304384 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.304504 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.306510 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.308003 2577 server.go:1019] "Starting client certificate rotation" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.308105 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.308137 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.310762 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.317197 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.333526 2577 log.go:25] "Validated CRI v1 runtime API" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.336002 2577 log.go:25] "Validated CRI v1 image API" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.336855 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 01 21:11:41.640810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.339418 2577 fs.go:135] Filesystem UUIDs: map[29b81805-058b-4339-a214-c9dc017c854a:/dev/nvme0n1p4 465ce23a-0c44-4dbb-9756-db61c097b5fc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 01 21:11:41.641320 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.339435 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.351809 2577 manager.go:216] Machine: {Timestamp:2026-07-01 21:11:41.350488572 +0000 UTC m=+0.405020645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100817 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20f9b541332650d596ad0e32517b52 SystemUUID:ec20f9b5-4133-2650-d596-ad0e32517b52 BootID:03b75ac0-83e1-4966-94d1-78a5e87ccec0 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:96:c1:aa:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:96:c1:aa:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:20:f8:ae:83:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.351909 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.352032 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.354684 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.354718 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.355357 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.355368 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 01 21:11:41.641403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.355375 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.355509 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.362221 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.363130 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366014 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366125 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366153 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366162 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366190 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366200 2577 kubelet.go:398] "Adding apiserver pod source" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366210 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366520 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.366546 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.368745 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.370445 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.370489 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.370745 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f4ww" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371792 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371815 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371824 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371832 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371844 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371852 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371863 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371871 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371882 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371891 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371903 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.371914 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.372855 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.372866 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.375725 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.375732 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.375746 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.378315 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f4ww" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.379283 2577 server.go:1268] "Started kubelet" Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.379357 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.379481 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 01 21:11:41.641744 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.379535 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.380819 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.381749 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.383951 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-112.ec2.internal.18be46c1f7703a75 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-112.ec2.internal,UID:ip-10-0-140-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-112.ec2.internal,},FirstTimestamp:2026-07-01 21:11:41.379254901 +0000 UTC m=+0.433786975,LastTimestamp:2026-07-01 21:11:41.379254901 +0000 UTC m=+0.433786975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-112.ec2.internal,}" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.384887 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.386911 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.386927 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.386990 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.388071 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.388088 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.389560 2577 factory.go:55] Registering systemd factory Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.389583 2577 factory.go:223] Registration of the systemd container factory successfully Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.389758 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.389863 2577 factory.go:153] Registering CRI-O factory Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.389880 2577 factory.go:223] Registration of the crio container factory successfully Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.390001 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.390025 2577 factory.go:103] Registering Raw factory Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.390041 2577 manager.go:1196] Started watching for new ooms in manager Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.390506 2577 manager.go:349] Starting recovery of all containers Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.397989 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.398000 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.404971 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.408261 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-112.ec2.internal\" not found" node="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.409871 2577 manager.go:354] Recovery completed Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.411483 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.414885 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.416786 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.416819 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.416832 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.417382 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.417394 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.417413 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.417411 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418038 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.643263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418058 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418067 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418771 2577 policy_none.go:49] "None policy: Start" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418783 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.418793 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.420032 2577 policy_none.go:47] "Start" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.473951 2577 manager.go:337] "Starting Device Plugin manager" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.474202 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474217 2577 server.go:87] "Starting device plugin registration server" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474475 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474484 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474588 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474669 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.474678 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.475278 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.475331 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.488687 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.489883 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.489912 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.489931 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.489964 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.492428 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.575366 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.581960 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.581985 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.581998 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.582021 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.588397 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.588418 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-112.ec2.internal\": node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.590540 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal"] Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.590593 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.592038 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.592061 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.592074 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.593596 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.593754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.593779 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594377 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.644238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594396 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594404 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594717 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594733 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.594742 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.595772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.595794 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.596415 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientMemory" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.596433 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.596441 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeHasSufficientPID" Jul 01 21:11:41.645264 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.602552 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.647099 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-112.ec2.internal\" not found" node="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.653248 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-112.ec2.internal\" not found" node="ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.700544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.700572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.700592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae7768ae7f0cea9300e6e78d611f1271-config\") pod \"kube-apiserver-proxy-ip-10-0-140-112.ec2.internal\" (UID: \"ae7768ae7f0cea9300e6e78d611f1271\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.777765 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.703604 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae7768ae7f0cea9300e6e78d611f1271-config\") pod \"kube-apiserver-proxy-ip-10-0-140-112.ec2.internal\" (UID: \"ae7768ae7f0cea9300e6e78d611f1271\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801662 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801722 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae7768ae7f0cea9300e6e78d611f1271-config\") pod \"kube-apiserver-proxy-ip-10-0-140-112.ec2.internal\" (UID: \"ae7768ae7f0cea9300e6e78d611f1271\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.801760 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b41e6be2d48d7bdf83317dba87027e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal\" (UID: \"1b41e6be2d48d7bdf83317dba87027e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.804640 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:41.916107 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:41.905416 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.042578 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.950569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:42.042578 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:41.959890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:42.042578 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.006487 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.174231 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.107006 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.313056 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.207523 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.313056 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.307079 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 01 21:11:42.313056 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.307256 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 01 21:11:42.313056 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.307262 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 01 21:11:42.313056 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.308133 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.388334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.382848 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 21:06:41 +0000 UTC" deadline="2028-01-08 21:38:26.457192548 +0000 UTC" Jul 01 21:11:42.388334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.382887 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13344h26m44.074307545s" Jul 01 21:11:42.473778 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.408349 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.591206 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.508690 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.694710 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:42.609584 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-112.ec2.internal\" not found" Jul 01 21:11:42.694710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.649045 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:42.694710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.661315 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:42.694710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.690047 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" Jul 01 21:11:42.704558 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.704533 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 01 21:11:42.705570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.705553 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" Jul 01 21:11:42.713454 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.713433 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 01 21:11:42.717740 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:42.717719 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b41e6be2d48d7bdf83317dba87027e0.slice/crio-c2da873d1372baac18bb5fca81d49aee06c6a8a669c593442f60ba7ef59874a6 WatchSource:0}: Error finding container c2da873d1372baac18bb5fca81d49aee06c6a8a669c593442f60ba7ef59874a6: Status 404 returned error can't find the container with id c2da873d1372baac18bb5fca81d49aee06c6a8a669c593442f60ba7ef59874a6 Jul 01 21:11:42.718304 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:42.718288 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7768ae7f0cea9300e6e78d611f1271.slice/crio-83b576f7b55ceabfea63cae0d0118407fafdc722e190d4970f7fdbfd6a4ffc4d WatchSource:0}: Error finding container 83b576f7b55ceabfea63cae0d0118407fafdc722e190d4970f7fdbfd6a4ffc4d: Status 404 returned error can't find the container with id 83b576f7b55ceabfea63cae0d0118407fafdc722e190d4970f7fdbfd6a4ffc4d Jul 01 21:11:42.730914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:42.730894 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 01 21:11:43.131808 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.131781 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:43.261119 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.261085 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 01 21:11:43.368141 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.368090 2577 apiserver.go:52] "Watching apiserver" Jul 01 21:11:43.373203 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.373003 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 01 21:11:43.373452 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.373433 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal","openshift-multus/network-metrics-daemon-q79ff","openshift-network-diagnostics/network-check-target-qnl77","openshift-network-operator/iptables-alerter-28nm5","openshift-ovn-kubernetes/ovnkube-node-dm7qs","kube-system/global-pull-secret-syncer-gl4jg","kube-system/konnectivity-agent-xxxwx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d","openshift-cluster-node-tuning-operator/tuned-2k679","openshift-dns/node-resolver-zc8cn","openshift-image-registry/node-ca-p6vwh","openshift-multus/multus-additional-cni-plugins-v9fxl","openshift-multus/multus-r96q2","kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal"] Jul 01 21:11:43.375966 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.375942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.378583 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.378562 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 01 21:11:43.378799 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.378785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d287r\"" Jul 01 21:11:43.379808 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.378958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.380480 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.380453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:43.380572 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.380550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:43.380958 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.380940 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.381526 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.381360 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 01 21:11:43.381526 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.381361 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 01 21:11:43.381526 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.381499 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-phn88\"" Jul 01 21:11:43.381717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.381624 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.381772 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.381730 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 01 21:11:43.382983 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.382631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.384838 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.384416 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.384838 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.384484 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nfr2g\"" Jul 01 21:11:43.384838 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.384617 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.384838 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.384675 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 01 21:11:43.385170 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.385150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.387912 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.387820 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 01 21:11:43.389093 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.389075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.389668 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.389322 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:43.390066 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.390044 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 01 21:11:43.390574 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.390553 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 01 21:11:43.390761 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.390745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.391634 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.391003 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 01 21:11:43.391634 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.391465 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.391817 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.391688 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.391967 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.391950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 01 21:11:43.392238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.392222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2hsts\"" Jul 01 21:11:43.394725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.394237 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.394725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.394539 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4sm7\"" Jul 01 21:11:43.394725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.394579 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.394899 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.394731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.396687 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.396668 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.396793 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.396767 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.396862 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.396791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4r9m6\"" Jul 01 21:11:43.396862 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.396844 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 01 21:11:43.397061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.397042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.397153 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.397114 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:43.399556 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.399537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.400992 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.400971 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 01 21:11:43.401422 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.401406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.401531 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.401510 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.401700 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.401675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9xp7q\"" Jul 01 21:11:43.402128 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.402095 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.402968 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.402949 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 01 21:11:43.403340 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.403320 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jptcq\"" Jul 01 21:11:43.403459 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.403443 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 01 21:11:43.403545 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.403528 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 01 21:11:43.404431 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.404412 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.406787 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.406743 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 01 21:11:43.406976 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.406960 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 01 21:11:43.407292 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.407273 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j2m97\"" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-ovn\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-host\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00ca11bc-d58b-4fbb-871b-f5de4c360041-host-slash\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-netns\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-node-log\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-modprobe-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j777x\" (UniqueName: \"kubernetes.io/projected/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-kube-api-access-j777x\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-bin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.409872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysconfig\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-kubernetes\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-conf\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-tmp\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.409980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-kubelet\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgwb\" (UniqueName: \"kubernetes.io/projected/bd66d419-d121-470c-a388-4d3b1ff97ea4-kube-api-access-zlgwb\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-etc-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-sys\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/288a741e-d8d9-444e-91ce-2a4481b0ed30-konnectivity-ca\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-netns\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-daemon-config\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-script-lib\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.410530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-multus-certs\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-bin\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-systemd\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-systemd-units\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-run\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/288a741e-d8d9-444e-91ce-2a4481b0ed30-agent-certs\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-system-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-os-release\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-multus\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-sys-fs\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-registration-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-var-lib-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-netd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-kubelet-config\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-log-socket\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-config\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7dp\" (UniqueName: \"kubernetes.io/projected/813b4b2e-b459-4809-b455-f29d00c476e0-kube-api-access-nv7dp\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-dbus\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-device-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-lib-modules\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.410992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-cnibin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-hostroot\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-conf-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzqp\" (UniqueName: \"kubernetes.io/projected/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kube-api-access-nhzqp\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-var-lib-kubelet\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.411892 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-socket-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00ca11bc-d58b-4fbb-871b-f5de4c360041-iptables-alerter-script\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-kubelet\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-systemd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-tuned\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-socket-dir-parent\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-k8s-cni-cncf-io\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-etc-kubernetes\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vxp\" (UniqueName: \"kubernetes.io/projected/00ca11bc-d58b-4fbb-871b-f5de4c360041-kube-api-access-49vxp\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-env-overrides\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/813b4b2e-b459-4809-b455-f29d00c476e0-ovn-node-metrics-cert\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-cni-binary-copy\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-slash\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.412571 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.411563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.422026 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.422004 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tql6j" Jul 01 21:11:43.429325 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.429307 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tql6j" Jul 01 21:11:43.490522 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.490487 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 01 21:11:43.502443 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.502380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" event={"ID":"ae7768ae7f0cea9300e6e78d611f1271","Type":"ContainerStarted","Data":"83b576f7b55ceabfea63cae0d0118407fafdc722e190d4970f7fdbfd6a4ffc4d"} Jul 01 21:11:43.505094 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.505059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" event={"ID":"1b41e6be2d48d7bdf83317dba87027e0","Type":"ContainerStarted","Data":"c2da873d1372baac18bb5fca81d49aee06c6a8a669c593442f60ba7ef59874a6"} Jul 01 21:11:43.512662 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-bin\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-systemd\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-host\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-systemd-units\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-run\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512761 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-bin\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/288a741e-d8d9-444e-91ce-2a4481b0ed30-agent-certs\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-systemd-units\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512773 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-systemd\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.512819 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-system-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-system-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-run\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-os-release\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-multus\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.512992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-system-cni-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-os-release\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-multus\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-sys-fs\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-registration-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-var-lib-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-netd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-cni-netd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513167 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513156 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-registration-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-sys-fs\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-var-lib-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-kubelet-config\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513269 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-kubelet-config\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-log-socket\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-config\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7dp\" (UniqueName: \"kubernetes.io/projected/813b4b2e-b459-4809-b455-f29d00c476e0-kube-api-access-nv7dp\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513330 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-log-socket\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-dbus\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-serviceca\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-device-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.513726 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55550433-0881-4a26-b1e8-13b7b9e6e937-dbus\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-lib-modules\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-cnibin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-hostroot\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-conf-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-conf-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-os-release\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgb2\" (UniqueName: \"kubernetes.io/projected/ab94aa7d-dcdf-4038-a02d-eb23db442513-kube-api-access-2mgb2\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513615 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-cnibin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-device-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-hostroot\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzqp\" (UniqueName: \"kubernetes.io/projected/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kube-api-access-nhzqp\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.514351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-var-lib-kubelet\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d70855-fe25-4e82-9b5f-694d5890e5be-hosts-file\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513781 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-lib-modules\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-socket-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00ca11bc-d58b-4fbb-871b-f5de4c360041-iptables-alerter-script\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-kubelet\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-systemd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-tuned\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-socket-dir-parent\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-var-lib-kubelet\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-k8s-cni-cncf-io\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-etc-kubernetes\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vxp\" (UniqueName: \"kubernetes.io/projected/00ca11bc-d58b-4fbb-871b-f5de4c360041-kube-api-access-49vxp\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514012 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-kubelet\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514032 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-systemd\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-k8s-cni-cncf-io\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.513983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-env-overrides\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/813b4b2e-b459-4809-b455-f29d00c476e0-ovn-node-metrics-cert\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-cni-binary-copy\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-socket-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxv6\" (UniqueName: \"kubernetes.io/projected/a911bcd8-9282-410b-8039-b9d043324899-kube-api-access-vhxv6\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7qg\" (UniqueName: \"kubernetes.io/projected/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-kube-api-access-hf7qg\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-slash\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-ovn\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.515723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514367 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-config\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-host\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-cnibin\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514422 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-socket-dir-parent\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00ca11bc-d58b-4fbb-871b-f5de4c360041-host-slash\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-netns\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-etc-kubernetes\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-node-log\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514526 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00ca11bc-d58b-4fbb-871b-f5de4c360041-iptables-alerter-script\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-modprobe-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514575 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-env-overrides\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j777x\" (UniqueName: \"kubernetes.io/projected/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-kube-api-access-j777x\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.516351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514661 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-slash\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-host\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514752 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-cni-dir\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-run-ovn\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514859 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00ca11bc-d58b-4fbb-871b-f5de4c360041-host-slash\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514919 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.514964 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-node-log\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-netns\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-cni-binary-copy\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-modprobe-d\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-bin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-cni-bin\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysconfig\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-kubernetes\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515317 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-kubernetes\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-conf\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-tmp\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-kubelet\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgwb\" (UniqueName: \"kubernetes.io/projected/bd66d419-d121-470c-a388-4d3b1ff97ea4-kube-api-access-zlgwb\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d70855-fe25-4e82-9b5f-694d5890e5be-tmp-dir\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-etc-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-sys\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/288a741e-d8d9-444e-91ce-2a4481b0ed30-konnectivity-ca\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-netns\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.517713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-daemon-config\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8tm\" (UniqueName: \"kubernetes.io/projected/b8d70855-fe25-4e82-9b5f-694d5890e5be-kube-api-access-lh8tm\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-script-lib\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-multus-certs\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.515887 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-multus-certs\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.516035 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.516102 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:44.016082212 +0000 UTC m=+3.070614278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-sys\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/813b4b2e-b459-4809-b455-f29d00c476e0-etc-openvswitch\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-run-netns\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516432 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd66d419-d121-470c-a388-4d3b1ff97ea4-multus-daemon-config\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516509 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysconfig\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-sysctl-conf\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.518386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516622 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd66d419-d121-470c-a388-4d3b1ff97ea4-host-var-lib-kubelet\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.519069 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.516858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/288a741e-d8d9-444e-91ce-2a4481b0ed30-konnectivity-ca\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.519069 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.517114 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/813b4b2e-b459-4809-b455-f29d00c476e0-ovnkube-script-lib\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.520222 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.520200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-tmp\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.520510 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.520462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/288a741e-d8d9-444e-91ce-2a4481b0ed30-agent-certs\") pod \"konnectivity-agent-xxxwx\" (UID: \"288a741e-d8d9-444e-91ce-2a4481b0ed30\") " pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.521100 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.520807 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-etc-tuned\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.521100 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.520818 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:43.521100 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.520836 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:43.521100 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.520849 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:43.521100 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.520906 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:44.020885817 +0000 UTC m=+3.075417877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:43.522168 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.522121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/813b4b2e-b459-4809-b455-f29d00c476e0-ovn-node-metrics-cert\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.523650 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.523587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7dp\" (UniqueName: \"kubernetes.io/projected/813b4b2e-b459-4809-b455-f29d00c476e0-kube-api-access-nv7dp\") pod \"ovnkube-node-dm7qs\" (UID: \"813b4b2e-b459-4809-b455-f29d00c476e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.523815 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.523794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzqp\" (UniqueName: \"kubernetes.io/projected/a0a62b0f-258f-4e09-8af4-6c10e92a7bf6-kube-api-access-nhzqp\") pod \"aws-ebs-csi-driver-node-nnh5d\" (UID: \"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.525034 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.525004 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vxp\" (UniqueName: \"kubernetes.io/projected/00ca11bc-d58b-4fbb-871b-f5de4c360041-kube-api-access-49vxp\") pod \"iptables-alerter-28nm5\" (UID: \"00ca11bc-d58b-4fbb-871b-f5de4c360041\") " pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.526382 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.526358 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j777x\" (UniqueName: \"kubernetes.io/projected/3ef7f221-0368-47c2-8f90-a7ad55fd4ce5-kube-api-access-j777x\") pod \"tuned-2k679\" (UID: \"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5\") " pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.526468 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.526395 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgwb\" (UniqueName: \"kubernetes.io/projected/bd66d419-d121-470c-a388-4d3b1ff97ea4-kube-api-access-zlgwb\") pod \"multus-r96q2\" (UID: \"bd66d419-d121-470c-a388-4d3b1ff97ea4\") " pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.617125 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d70855-fe25-4e82-9b5f-694d5890e5be-tmp-dir\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8tm\" (UniqueName: \"kubernetes.io/projected/b8d70855-fe25-4e82-9b5f-694d5890e5be-kube-api-access-lh8tm\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-host\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-system-cni-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-serviceca\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-os-release\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgb2\" (UniqueName: \"kubernetes.io/projected/ab94aa7d-dcdf-4038-a02d-eb23db442513-kube-api-access-2mgb2\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d70855-fe25-4e82-9b5f-694d5890e5be-hosts-file\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxv6\" (UniqueName: \"kubernetes.io/projected/a911bcd8-9282-410b-8039-b9d043324899-kube-api-access-vhxv6\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7qg\" (UniqueName: \"kubernetes.io/projected/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-kube-api-access-hf7qg\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-cnibin\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.617771 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.617551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618142 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-os-release\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618225 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618144 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618225 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d70855-fe25-4e82-9b5f-694d5890e5be-hosts-file\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.618337 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-cnibin\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618337 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab94aa7d-dcdf-4038-a02d-eb23db442513-system-cni-dir\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618611 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d70855-fe25-4e82-9b5f-694d5890e5be-tmp-dir\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.618670 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.618733 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618680 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-host\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.618733 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.618683 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:43.618833 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:43.618741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:44.118723764 +0000 UTC m=+3.173255831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:43.618833 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.618756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab94aa7d-dcdf-4038-a02d-eb23db442513-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.619049 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.619027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-serviceca\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.626800 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.626758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxv6\" (UniqueName: \"kubernetes.io/projected/a911bcd8-9282-410b-8039-b9d043324899-kube-api-access-vhxv6\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:43.626923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.626896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8tm\" (UniqueName: \"kubernetes.io/projected/b8d70855-fe25-4e82-9b5f-694d5890e5be-kube-api-access-lh8tm\") pod \"node-resolver-zc8cn\" (UID: \"b8d70855-fe25-4e82-9b5f-694d5890e5be\") " pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.628940 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.628916 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7qg\" (UniqueName: \"kubernetes.io/projected/c9b97c52-11f1-4a3e-9d3c-f75126b58de3-kube-api-access-hf7qg\") pod \"node-ca-p6vwh\" (UID: \"c9b97c52-11f1-4a3e-9d3c-f75126b58de3\") " pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.629048 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.628925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgb2\" (UniqueName: \"kubernetes.io/projected/ab94aa7d-dcdf-4038-a02d-eb23db442513-kube-api-access-2mgb2\") pod \"multus-additional-cni-plugins-v9fxl\" (UID: \"ab94aa7d-dcdf-4038-a02d-eb23db442513\") " pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:43.690738 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.690668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:11:43.703837 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.703805 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r96q2" Jul 01 21:11:43.716501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.716479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28nm5" Jul 01 21:11:43.721308 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.721291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:11:43.730093 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.730072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2k679" Jul 01 21:11:43.738698 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.738678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" Jul 01 21:11:43.744406 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.744379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zc8cn" Jul 01 21:11:43.759007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.758986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6vwh" Jul 01 21:11:43.764617 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:43.764601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" Jul 01 21:11:44.021293 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:44.021201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:44.021293 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:44.021249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:44.021484 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021371 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:44.021484 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021399 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:44.021484 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021427 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:44.021484 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021437 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:45.02141991 +0000 UTC m=+4.075951972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:44.021484 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021442 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:44.021713 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.021504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:45.021486335 +0000 UTC m=+4.076018397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:44.122503 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:44.122465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:44.122672 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.122602 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:44.122672 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:44.122671 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:45.122654534 +0000 UTC m=+4.177186618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:44.433417 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:44.433311 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 21:06:43 +0000 UTC" deadline="2028-02-07 05:34:22.017939853 +0000 UTC" Jul 01 21:11:44.433417 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:44.433367 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14048h22m37.584576962s" Jul 01 21:11:44.567411 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.567384 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813b4b2e_b459_4809_b455_f29d00c476e0.slice/crio-1c44731ef6db9ad57a3bfcfc2df85a8c4cba01a896ec15b9b8f360dc8733917e WatchSource:0}: Error finding container 1c44731ef6db9ad57a3bfcfc2df85a8c4cba01a896ec15b9b8f360dc8733917e: Status 404 returned error can't find the container with id 1c44731ef6db9ad57a3bfcfc2df85a8c4cba01a896ec15b9b8f360dc8733917e Jul 01 21:11:44.571149 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.571106 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b97c52_11f1_4a3e_9d3c_f75126b58de3.slice/crio-0bdb8562a71eff0111aec01681ce6725b8642ffd256f259c5abed5c62b50082d WatchSource:0}: Error finding container 0bdb8562a71eff0111aec01681ce6725b8642ffd256f259c5abed5c62b50082d: Status 404 returned error can't find the container with id 0bdb8562a71eff0111aec01681ce6725b8642ffd256f259c5abed5c62b50082d Jul 01 21:11:44.572329 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.572303 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a741e_d8d9_444e_91ce_2a4481b0ed30.slice/crio-596d54d3e0d691861747436e2a74217a50abed0bd310ed7b2cb8965cf47d9e11 WatchSource:0}: Error finding container 596d54d3e0d691861747436e2a74217a50abed0bd310ed7b2cb8965cf47d9e11: Status 404 returned error can't find the container with id 596d54d3e0d691861747436e2a74217a50abed0bd310ed7b2cb8965cf47d9e11 Jul 01 21:11:44.573080 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.573060 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d70855_fe25_4e82_9b5f_694d5890e5be.slice/crio-c32fde55dd08dfac080f84933644ac4e95a912a7d68cfd7b2bb83d127a2e95ab WatchSource:0}: Error finding container c32fde55dd08dfac080f84933644ac4e95a912a7d68cfd7b2bb83d127a2e95ab: Status 404 returned error can't find the container with id c32fde55dd08dfac080f84933644ac4e95a912a7d68cfd7b2bb83d127a2e95ab Jul 01 21:11:44.574327 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.574306 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef7f221_0368_47c2_8f90_a7ad55fd4ce5.slice/crio-c3209c76a4c22f3347c9207cad22eec392434384d37f9e93153bc07f4555ac66 WatchSource:0}: Error finding container c3209c76a4c22f3347c9207cad22eec392434384d37f9e93153bc07f4555ac66: Status 404 returned error can't find the container with id c3209c76a4c22f3347c9207cad22eec392434384d37f9e93153bc07f4555ac66 Jul 01 21:11:44.576576 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.576058 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a62b0f_258f_4e09_8af4_6c10e92a7bf6.slice/crio-d227f7002ec7e403f59f11bdb88684ea354ef3e77f9e005cae88a704b3cbe9f6 WatchSource:0}: Error finding container d227f7002ec7e403f59f11bdb88684ea354ef3e77f9e005cae88a704b3cbe9f6: Status 404 returned error can't find the container with id d227f7002ec7e403f59f11bdb88684ea354ef3e77f9e005cae88a704b3cbe9f6 Jul 01 21:11:44.578308 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.578287 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab94aa7d_dcdf_4038_a02d_eb23db442513.slice/crio-8100ce271860a96e5a5d36a7a58925217452d4feb8d5742dd769948be8c541ae WatchSource:0}: Error finding container 8100ce271860a96e5a5d36a7a58925217452d4feb8d5742dd769948be8c541ae: Status 404 returned error can't find the container with id 8100ce271860a96e5a5d36a7a58925217452d4feb8d5742dd769948be8c541ae Jul 01 21:11:44.600530 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.600489 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd66d419_d121_470c_a388_4d3b1ff97ea4.slice/crio-d4d58783886553b92285139f195b9b3501a9ae4e3e832986209949bf06ddd4c7 WatchSource:0}: Error finding container d4d58783886553b92285139f195b9b3501a9ae4e3e832986209949bf06ddd4c7: Status 404 returned error can't find the container with id d4d58783886553b92285139f195b9b3501a9ae4e3e832986209949bf06ddd4c7 Jul 01 21:11:44.602326 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:11:44.602303 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ca11bc_d58b_4fbb_871b_f5de4c360041.slice/crio-e75f4651259f5b3555d156d675b81e781b670b6f62b0325a6d585aa79ec70925 WatchSource:0}: Error finding container e75f4651259f5b3555d156d675b81e781b670b6f62b0325a6d585aa79ec70925: Status 404 returned error can't find the container with id e75f4651259f5b3555d156d675b81e781b670b6f62b0325a6d585aa79ec70925 Jul 01 21:11:45.028681 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.028409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:45.028681 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028662 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:45.028681 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028692 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:45.028952 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028706 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:45.028952 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.028715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:45.028952 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028848 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:45.028952 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:47.028751534 +0000 UTC m=+6.083283600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:45.028952 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.028902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:47.028885951 +0000 UTC m=+6.083418018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:45.129801 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.129217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:45.129801 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.129368 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:45.129801 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.129436 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:47.129415613 +0000 UTC m=+6.183947676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:45.434577 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.434454 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 21:06:43 +0000 UTC" deadline="2027-12-05 18:50:24.31066054 +0000 UTC" Jul 01 21:11:45.434577 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.434494 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12525h38m38.876170652s" Jul 01 21:11:45.492554 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.492050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:45.492554 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.492202 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:45.492554 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.492321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:45.492554 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.492390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:45.492554 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.492492 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:45.492914 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:45.492580 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:45.521050 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.521013 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b41e6be2d48d7bdf83317dba87027e0" containerID="7c1ac25f0a6acdddf44a7944be113f75bd3660dfcf10d14e1aad33abc0abb210" exitCode=0 Jul 01 21:11:45.521240 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.521105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" event={"ID":"1b41e6be2d48d7bdf83317dba87027e0","Type":"ContainerDied","Data":"7c1ac25f0a6acdddf44a7944be113f75bd3660dfcf10d14e1aad33abc0abb210"} Jul 01 21:11:45.525466 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.525407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" event={"ID":"ae7768ae7f0cea9300e6e78d611f1271","Type":"ContainerStarted","Data":"d7b0edd0ce9efc9790f7f6f8b6f79cfdcd65c0c75c551251f1d7323732c0d46a"} Jul 01 21:11:45.532686 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.532613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r96q2" event={"ID":"bd66d419-d121-470c-a388-4d3b1ff97ea4","Type":"ContainerStarted","Data":"d4d58783886553b92285139f195b9b3501a9ae4e3e832986209949bf06ddd4c7"} Jul 01 21:11:45.538736 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.537030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2k679" event={"ID":"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5","Type":"ContainerStarted","Data":"c3209c76a4c22f3347c9207cad22eec392434384d37f9e93153bc07f4555ac66"} Jul 01 21:11:45.543556 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.543529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xxxwx" event={"ID":"288a741e-d8d9-444e-91ce-2a4481b0ed30","Type":"ContainerStarted","Data":"596d54d3e0d691861747436e2a74217a50abed0bd310ed7b2cb8965cf47d9e11"} Jul 01 21:11:45.552564 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.552285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6vwh" event={"ID":"c9b97c52-11f1-4a3e-9d3c-f75126b58de3","Type":"ContainerStarted","Data":"0bdb8562a71eff0111aec01681ce6725b8642ffd256f259c5abed5c62b50082d"} Jul 01 21:11:45.552564 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.552395 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-112.ec2.internal" podStartSLOduration=3.5523767790000003 podStartE2EDuration="3.552376779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:45.55145303 +0000 UTC m=+4.605985113" watchObservedRunningTime="2026-07-01 21:11:45.552376779 +0000 UTC m=+4.606908910" Jul 01 21:11:45.561795 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.561762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"1c44731ef6db9ad57a3bfcfc2df85a8c4cba01a896ec15b9b8f360dc8733917e"} Jul 01 21:11:45.565647 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.565619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28nm5" event={"ID":"00ca11bc-d58b-4fbb-871b-f5de4c360041","Type":"ContainerStarted","Data":"e75f4651259f5b3555d156d675b81e781b670b6f62b0325a6d585aa79ec70925"} Jul 01 21:11:45.567567 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.567531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerStarted","Data":"8100ce271860a96e5a5d36a7a58925217452d4feb8d5742dd769948be8c541ae"} Jul 01 21:11:45.587374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.582434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" event={"ID":"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6","Type":"ContainerStarted","Data":"d227f7002ec7e403f59f11bdb88684ea354ef3e77f9e005cae88a704b3cbe9f6"} Jul 01 21:11:45.588305 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:45.588254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zc8cn" event={"ID":"b8d70855-fe25-4e82-9b5f-694d5890e5be","Type":"ContainerStarted","Data":"c32fde55dd08dfac080f84933644ac4e95a912a7d68cfd7b2bb83d127a2e95ab"} Jul 01 21:11:46.595772 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:46.595701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" event={"ID":"1b41e6be2d48d7bdf83317dba87027e0","Type":"ContainerStarted","Data":"b639a0f75f31c8b2adfac4d7d8a4308441e4d582c6a2933f42fa14b26b6f95a5"} Jul 01 21:11:46.610953 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:46.609914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-112.ec2.internal" podStartSLOduration=4.60989107 podStartE2EDuration="4.60989107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:46.609679625 +0000 UTC m=+5.664211710" watchObservedRunningTime="2026-07-01 21:11:46.60989107 +0000 UTC m=+5.664423154" Jul 01 21:11:47.047840 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.047743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:47.047840 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.047808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:47.048061 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.047933 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:47.048061 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.048001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:51.047983141 +0000 UTC m=+10.102515214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:47.048527 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.048473 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:47.048527 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.048495 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:47.048527 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.048509 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:47.048689 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.048559 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:51.048543446 +0000 UTC m=+10.103075512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:47.148191 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.148131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:47.148382 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.148331 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:47.148442 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.148404 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:51.14838506 +0000 UTC m=+10.202917131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:47.492420 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.492338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:47.492420 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.492387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:47.492642 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.492461 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:47.492642 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.492530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:47.492642 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:47.492587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:47.492783 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:47.492655 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:49.490506 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:49.490436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:49.490923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:49.490436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:49.490923 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:49.490696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:49.490923 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:49.490578 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:49.490923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:49.490733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:49.490923 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:49.490817 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.076414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.076486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076612 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:59.076665386 +0000 UTC m=+18.131197452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076684 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076708 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076720 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:51.076781 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.076756 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:59.076746481 +0000 UTC m=+18.131278557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:51.177423 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.176784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:51.177423 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.176979 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:51.177423 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.177050 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:11:59.177029561 +0000 UTC m=+18.231561624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:51.491430 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.491355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:51.493286 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.492612 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:51.493286 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.493003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:51.493286 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.493107 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:51.493286 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:51.493163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:51.493286 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:51.493252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:53.490907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:53.490477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:53.490907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:53.490582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:53.490907 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:53.490587 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:53.490907 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:53.490721 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:53.490907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:53.490794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:53.491432 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:53.490929 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:55.490948 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:55.490843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:55.490948 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:55.490884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:55.490948 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:55.490881 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:55.491495 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:55.490981 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:55.491495 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:55.491086 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:55.491495 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:55.491225 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:57.490646 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:57.490606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:57.491101 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:57.490613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:57.491101 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:57.490765 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:57.491101 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:57.490812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:57.491101 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:57.490919 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:11:57.491101 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:57.491010 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:59.137388 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.137355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.137433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137509 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137530 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137543 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137555 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137579 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:15.137559736 +0000 UTC m=+34.192091819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:11:59.137806 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.137596 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:15.137587521 +0000 UTC m=+34.192119586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:11:59.238035 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.237997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:59.238210 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.238151 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:59.238273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.238233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:15.238216022 +0000 UTC m=+34.292748083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:11:59.491077 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.490943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:11:59.491077 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.490983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:11:59.491077 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:11:59.490948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:11:59.491342 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.491081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:11:59.491342 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.491217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:11:59.491342 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:11:59.491272 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:01.491602 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:01.491558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:01.492036 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:01.491720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:01.492036 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:01.491772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:01.492578 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:01.492497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:01.492958 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:01.492897 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:01.493125 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:01.493102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:03.494539 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.493001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:03.495409 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.493935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:03.495409 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.493972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:03.495409 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:03.495361 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:03.496405 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:03.495690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:03.496405 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:03.495849 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:03.631566 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.631284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zc8cn" event={"ID":"b8d70855-fe25-4e82-9b5f-694d5890e5be","Type":"ContainerStarted","Data":"a352095710b62c8e8095f1c812f2047d057c3a2fedbdaea4e05e17617f33d92e"} Jul 01 21:12:03.632873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.632835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r96q2" event={"ID":"bd66d419-d121-470c-a388-4d3b1ff97ea4","Type":"ContainerStarted","Data":"34e5cd28bf6f4f652875760d57e1926e4a2d3df0f927aa80fcd89c73d9db0a1d"} Jul 01 21:12:03.634467 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.634438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2k679" event={"ID":"3ef7f221-0368-47c2-8f90-a7ad55fd4ce5","Type":"ContainerStarted","Data":"2e80ad6ea492fb0b9da7f8811405d7837f78673275f461bc6ddecfe426b26216"} Jul 01 21:12:03.635852 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.635820 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xxxwx" event={"ID":"288a741e-d8d9-444e-91ce-2a4481b0ed30","Type":"ContainerStarted","Data":"90f34945c751c8321bfa72b6841a9a1ef74187162771e37b9ed16159b2924624"} Jul 01 21:12:03.637531 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.637507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6vwh" event={"ID":"c9b97c52-11f1-4a3e-9d3c-f75126b58de3","Type":"ContainerStarted","Data":"550d011a1166bd7426c9e075aaeb943d07b6f0a20dadf1a32e16d186f2ff9ce8"} Jul 01 21:12:03.639550 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.639529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"eb8bdc61f0da788e062a2adb2c0087391fd5ab10739baf1e0869e05783af0b9d"} Jul 01 21:12:03.641301 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.641275 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="8402fcbbe5a5d8f78345f01de4407f4bcef7cc56bcb80fa78234b5e8bfc9be8c" exitCode=0 Jul 01 21:12:03.641424 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.641355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"8402fcbbe5a5d8f78345f01de4407f4bcef7cc56bcb80fa78234b5e8bfc9be8c"} Jul 01 21:12:03.643012 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.642991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" event={"ID":"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6","Type":"ContainerStarted","Data":"acf7033bcfe767111cb82b1377373786ed091531df9f38da02576b07ad4700c3"} Jul 01 21:12:03.647304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.647264 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zc8cn" podStartSLOduration=3.932639465 podStartE2EDuration="22.647252307s" totalImagesPullingTime="18.714612842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:03.646679239 +0000 UTC m=+22.701211332" watchObservedRunningTime="2026-07-01 21:12:03.647252307 +0000 UTC m=+22.701784389" Jul 01 21:12:03.661384 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.661342 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2k679" podStartSLOduration=3.9229450359999998 podStartE2EDuration="22.661328123s" totalImagesPullingTime="18.738383087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:03.661069665 +0000 UTC m=+22.715601748" watchObservedRunningTime="2026-07-01 21:12:03.661328123 +0000 UTC m=+22.715860205" Jul 01 21:12:03.674020 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.673978 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6vwh" podStartSLOduration=3.956709037 podStartE2EDuration="22.67396322s" totalImagesPullingTime="18.717254183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:03.673463981 +0000 UTC m=+22.727996064" watchObservedRunningTime="2026-07-01 21:12:03.67396322 +0000 UTC m=+22.728495300" Jul 01 21:12:03.685871 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.685831 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xxxwx" podStartSLOduration=8.2284261 podStartE2EDuration="22.685821547s" totalImagesPullingTime="14.457395447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:03.685515304 +0000 UTC m=+22.740047386" watchObservedRunningTime="2026-07-01 21:12:03.685821547 +0000 UTC m=+22.740353629" Jul 01 21:12:03.728125 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:03.728083 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r96q2" podStartSLOduration=3.985811254 podStartE2EDuration="22.728068778s" totalImagesPullingTime="18.742257524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:03.727644953 +0000 UTC m=+22.782177035" watchObservedRunningTime="2026-07-01 21:12:03.728068778 +0000 UTC m=+22.782600860" Jul 01 21:12:04.648967 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.648875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"f20b7190f9ddae6093438db7fb3d50ed89c21a2890ecda729ba11c8e75e43b55"} Jul 01 21:12:04.649696 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.649008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"c3e146dcebb1b6c0d74d3874fe3d51821526dbc5ca2085ef23dfce636b4a77d9"} Jul 01 21:12:04.649696 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.649057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"dffd1b13218d23ec441b39e1f01b1513f09617b77ef4a9991344eb41e110c419"} Jul 01 21:12:04.649696 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.649105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"84289a9bea1a16326d9770b519e1cb99699c0c4b15b98eefb447d2ed59426fe1"} Jul 01 21:12:04.649696 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.649153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"b5b3e935765aeede5493796c01d4294e94c93259ed3b02c8a8def86ea229a8de"} Jul 01 21:12:04.829048 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:04.829023 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 01 21:12:05.489238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.489096 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T21:12:04.829042867Z","UUID":"50874ec8-4458-413b-9db7-efeae5d5960f","Handler":null,"Name":"","Endpoint":""} Jul 01 21:12:05.490557 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.490530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:05.490710 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:05.490666 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:05.490710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.490687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:05.490867 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:05.490843 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:05.490980 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.490938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:05.491376 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.491216 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 01 21:12:05.491376 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.491250 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 01 21:12:05.491376 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:05.491324 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:05.653040 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.653007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28nm5" event={"ID":"00ca11bc-d58b-4fbb-871b-f5de4c360041","Type":"ContainerStarted","Data":"11d726bdf875937b219a278d07416ae7099301681e43e412da58c42a4b758244"} Jul 01 21:12:05.655453 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.655427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" event={"ID":"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6","Type":"ContainerStarted","Data":"fa41c55e0bf685767800eb571dbfad5c99424dcff6e841b4c1899cab57c148ce"} Jul 01 21:12:05.666645 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:05.666601 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-28nm5" podStartSLOduration=5.980215611 podStartE2EDuration="24.666589268s" totalImagesPullingTime="18.686373657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:05.666574171 +0000 UTC m=+24.721106252" watchObservedRunningTime="2026-07-01 21:12:05.666589268 +0000 UTC m=+24.721121349" Jul 01 21:12:06.659581 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:06.659541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" event={"ID":"a0a62b0f-258f-4e09-8af4-6c10e92a7bf6","Type":"ContainerStarted","Data":"e67be845804af0cc037adbcd0fb49d2e9143571b504ebfae47c5bee44b5bbd18"} Jul 01 21:12:06.663222 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:06.663190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"fc3362b5e9aaa3bef7d42558e6535e9e46bf73ba9f7c9bf5447783d86840da0e"} Jul 01 21:12:06.676844 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:06.676797 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnh5d" podStartSLOduration=4.557994442 podStartE2EDuration="25.676784581s" totalImagesPullingTime="21.118790139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:06.676009706 +0000 UTC m=+25.730541789" watchObservedRunningTime="2026-07-01 21:12:06.676784581 +0000 UTC m=+25.731316663" Jul 01 21:12:07.033706 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.033446 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:12:07.034148 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.034128 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:12:07.491025 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.490984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:07.491025 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.491022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:07.491302 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.490989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:07.491302 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:07.491103 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:07.491302 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:07.491189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:07.491302 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:07.491285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:07.665694 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.665665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:12:07.666146 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:07.665903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xxxwx" Jul 01 21:12:08.670699 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.670467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" event={"ID":"813b4b2e-b459-4809-b455-f29d00c476e0","Type":"ContainerStarted","Data":"91462c18ee5dea3d65b42b09a2fbfc4166a2138d0f49c6cc87e976a3607b26e7"} Jul 01 21:12:08.671492 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.670761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:08.672389 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.672365 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="f994501bb6e4d1c8ab2a0beba53d40ef364d8727b791bf5246dda2b52cbd0d83" exitCode=0 Jul 01 21:12:08.672523 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.672449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"f994501bb6e4d1c8ab2a0beba53d40ef364d8727b791bf5246dda2b52cbd0d83"} Jul 01 21:12:08.686633 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.686612 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:08.695004 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:08.694967 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" podStartSLOduration=8.909285333 podStartE2EDuration="27.694954468s" totalImagesPullingTime="18.785669135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:08.69474843 +0000 UTC m=+27.749280522" watchObservedRunningTime="2026-07-01 21:12:08.694954468 +0000 UTC m=+27.749486550" Jul 01 21:12:09.490899 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.490863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:09.491129 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.490865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:09.491129 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:09.490971 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:09.491129 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.490988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:09.491129 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:09.491081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:09.491324 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:09.491161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:09.674793 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.674760 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:09.675311 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.675285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:09.692373 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:09.691922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:10.360971 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.360934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnl77"] Jul 01 21:12:10.361151 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.361092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:10.361244 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:10.361170 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:10.363087 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.363065 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q79ff"] Jul 01 21:12:10.363225 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.363153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:10.363270 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:10.363246 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:10.370986 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.370959 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gl4jg"] Jul 01 21:12:10.371104 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.371065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:10.371209 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:10.371164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:10.678778 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.678745 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="db99cbf632773f13925d20894ec66d435e78ccf422158df2731672f103aac4ff" exitCode=0 Jul 01 21:12:10.679307 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:10.678841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"db99cbf632773f13925d20894ec66d435e78ccf422158df2731672f103aac4ff"} Jul 01 21:12:11.491294 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:11.491257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:11.491481 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:11.491358 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:12.490460 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:12.490402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:12.490901 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:12.490402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:12.490901 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:12.490563 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:12.490901 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:12.490827 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:12.686810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:12.686723 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="9875324e7a3b480629de993502b29ea78822e66f24f978da3af1f585d35d0802" exitCode=0 Jul 01 21:12:12.686810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:12.686781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"9875324e7a3b480629de993502b29ea78822e66f24f978da3af1f585d35d0802"} Jul 01 21:12:13.490913 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:13.490880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:13.491451 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:13.491028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:14.491108 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:14.491076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:14.491688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:14.491122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:14.491688 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:14.491226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:12:14.491688 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:14.491351 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnl77" podUID="c40564fb-5f5a-4698-83fe-e5d0bc2c1d58" Jul 01 21:12:15.158036 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:15.158004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:15.158036 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:15.158050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:15.158273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158169 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 01 21:12:15.158273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158206 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 01 21:12:15.158273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158227 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 01 21:12:15.158273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158238 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lxc64 for pod openshift-network-diagnostics/network-check-target-qnl77: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:12:15.158273 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret podName:55550433-0881-4a26-b1e8-13b7b9e6e937 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:47.158226396 +0000 UTC m=+66.212758461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret") pod "global-pull-secret-syncer-gl4jg" (UID: "55550433-0881-4a26-b1e8-13b7b9e6e937") : object "kube-system"/"original-pull-secret" not registered Jul 01 21:12:15.158457 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.158286 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64 podName:c40564fb-5f5a-4698-83fe-e5d0bc2c1d58 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:47.15826878 +0000 UTC m=+66.212800851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxc64" (UniqueName: "kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64") pod "network-check-target-qnl77" (UID: "c40564fb-5f5a-4698-83fe-e5d0bc2c1d58") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 01 21:12:15.259268 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:15.259239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:15.259425 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.259355 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:12:15.259425 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.259414 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:47.259399111 +0000 UTC m=+66.313931171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 01 21:12:15.490876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:15.490802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:15.491030 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:15.490944 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gl4jg" podUID="55550433-0881-4a26-b1e8-13b7b9e6e937" Jul 01 21:12:16.155227 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.155170 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-112.ec2.internal" event="NodeReady" Jul 01 21:12:16.155736 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.155355 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 01 21:12:16.194966 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.194934 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wncw6"] Jul 01 21:12:16.229185 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.229146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cvzw6"] Jul 01 21:12:16.229345 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.229285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.231647 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.231616 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 01 21:12:16.231647 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.231639 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 01 21:12:16.231858 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.231621 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmdpg\"" Jul 01 21:12:16.243091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.243069 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wncw6"] Jul 01 21:12:16.243226 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.243102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cvzw6"] Jul 01 21:12:16.243226 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.243171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.245869 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.245847 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 01 21:12:16.245992 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.245863 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 01 21:12:16.245992 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.245937 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dtznk\"" Jul 01 21:12:16.246221 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.246207 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 01 21:12:16.366840 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.366790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a87629f6-6a84-4c9e-9a9d-730de62f9df1-cert\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.366840 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.366841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5557a0e-5758-4f66-951b-8b7d55673f07-tmp-dir\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.367071 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.366903 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.367071 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.366939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j25zh\" (UniqueName: \"kubernetes.io/projected/b5557a0e-5758-4f66-951b-8b7d55673f07-kube-api-access-j25zh\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.367071 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.367023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p8p\" (UniqueName: \"kubernetes.io/projected/a87629f6-6a84-4c9e-9a9d-730de62f9df1-kube-api-access-w4p8p\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.367071 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.367043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5557a0e-5758-4f66-951b-8b7d55673f07-config-volume\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.468450 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p8p\" (UniqueName: \"kubernetes.io/projected/a87629f6-6a84-4c9e-9a9d-730de62f9df1-kube-api-access-w4p8p\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.468450 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5557a0e-5758-4f66-951b-8b7d55673f07-config-volume\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.468688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a87629f6-6a84-4c9e-9a9d-730de62f9df1-cert\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.468688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5557a0e-5758-4f66-951b-8b7d55673f07-tmp-dir\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.468688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.468688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j25zh\" (UniqueName: \"kubernetes.io/projected/b5557a0e-5758-4f66-951b-8b7d55673f07-kube-api-access-j25zh\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.468953 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:16.468927 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:16.469045 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:16.469001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:16.968981612 +0000 UTC m=+36.023513691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:16.469045 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.469006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5557a0e-5758-4f66-951b-8b7d55673f07-config-volume\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.469045 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.468933 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5557a0e-5758-4f66-951b-8b7d55673f07-tmp-dir\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.473893 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.473861 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a87629f6-6a84-4c9e-9a9d-730de62f9df1-cert\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.481011 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.480960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j25zh\" (UniqueName: \"kubernetes.io/projected/b5557a0e-5758-4f66-951b-8b7d55673f07-kube-api-access-j25zh\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.481126 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.481051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p8p\" (UniqueName: \"kubernetes.io/projected/a87629f6-6a84-4c9e-9a9d-730de62f9df1-kube-api-access-w4p8p\") pod \"ingress-canary-cvzw6\" (UID: \"a87629f6-6a84-4c9e-9a9d-730de62f9df1\") " pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.490840 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.490811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:16.491192 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.491003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:16.493061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.493043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 01 21:12:16.493383 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.493339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 01 21:12:16.493383 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.493362 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 01 21:12:16.493540 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.493415 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2xqn\"" Jul 01 21:12:16.493540 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.493365 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xqnfx\"" Jul 01 21:12:16.553259 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.553228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cvzw6" Jul 01 21:12:16.971874 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:16.971846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:16.972052 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:16.972016 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:16.972099 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:16.972092 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:17.972072735 +0000 UTC m=+37.026604816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:17.350345 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.350127 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cvzw6"] Jul 01 21:12:17.385142 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:12:17.385103 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87629f6_6a84_4c9e_9a9d_730de62f9df1.slice/crio-2178e38b5bf23ae402888e0db149159e4bfc42ff926518e5412f3a94a293e4e0 WatchSource:0}: Error finding container 2178e38b5bf23ae402888e0db149159e4bfc42ff926518e5412f3a94a293e4e0: Status 404 returned error can't find the container with id 2178e38b5bf23ae402888e0db149159e4bfc42ff926518e5412f3a94a293e4e0 Jul 01 21:12:17.490680 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.490665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:17.492795 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.492775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 01 21:12:17.703549 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.703516 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="af65b927a82e96ed3c4b0eef783a9a905b9b1ebe9152dbca01076c4003f4df83" exitCode=0 Jul 01 21:12:17.703713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.703601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"af65b927a82e96ed3c4b0eef783a9a905b9b1ebe9152dbca01076c4003f4df83"} Jul 01 21:12:17.704671 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.704651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cvzw6" event={"ID":"a87629f6-6a84-4c9e-9a9d-730de62f9df1","Type":"ContainerStarted","Data":"2178e38b5bf23ae402888e0db149159e4bfc42ff926518e5412f3a94a293e4e0"} Jul 01 21:12:17.981545 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:17.981513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:17.981760 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:17.981677 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:17.981760 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:17.981743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:19.981726671 +0000 UTC m=+39.036258734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:18.709305 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:18.709271 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab94aa7d-dcdf-4038-a02d-eb23db442513" containerID="25dda3fcc5dbaa5495211f079da29225edd3921330020ffdb013c8b9915af735" exitCode=0 Jul 01 21:12:18.709729 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:18.709315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerDied","Data":"25dda3fcc5dbaa5495211f079da29225edd3921330020ffdb013c8b9915af735"} Jul 01 21:12:19.715899 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:19.715855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" event={"ID":"ab94aa7d-dcdf-4038-a02d-eb23db442513","Type":"ContainerStarted","Data":"a94eb2f3cf154249612d7e6d161f3b178bf28da8826925324270da76d1589813"} Jul 01 21:12:19.737073 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:19.737018 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v9fxl" podStartSLOduration=8.151929376 podStartE2EDuration="38.736999486s" totalImagesPullingTime="30.58507011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:19.735943994 +0000 UTC m=+38.790476075" watchObservedRunningTime="2026-07-01 21:12:19.736999486 +0000 UTC m=+38.791531569" Jul 01 21:12:19.996162 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:19.995979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:19.996280 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:19.996112 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:19.996280 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:19.996243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:23.996227279 +0000 UTC m=+43.050759358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:20.720170 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:20.720132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cvzw6" event={"ID":"a87629f6-6a84-4c9e-9a9d-730de62f9df1","Type":"ContainerStarted","Data":"d945e252e437dd29d01551fe681095fe8737ed391fada313734afa8a2aee7575"} Jul 01 21:12:20.736054 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:20.736011 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cvzw6" podStartSLOduration=2.16935601 podStartE2EDuration="4.735997327s" totalImagesPullingTime="2.566641317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:20.73492091 +0000 UTC m=+39.789452993" watchObservedRunningTime="2026-07-01 21:12:20.735997327 +0000 UTC m=+39.790529409" Jul 01 21:12:24.023083 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:24.023040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:24.023456 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:24.023215 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:24.023456 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:24.023281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:32.023265425 +0000 UTC m=+51.077797485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:32.077689 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:32.077647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:32.078222 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:32.077797 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:32.078222 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:32.077861 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:12:48.077844046 +0000 UTC m=+67.132376106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:41.691720 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:41.691689 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dm7qs" Jul 01 21:12:47.187131 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.187093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:47.187131 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.187139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:47.189370 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.189352 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 01 21:12:47.189425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.189387 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 01 21:12:47.199835 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.199809 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 01 21:12:47.201729 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.201709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55550433-0881-4a26-b1e8-13b7b9e6e937-original-pull-secret\") pod \"global-pull-secret-syncer-gl4jg\" (UID: \"55550433-0881-4a26-b1e8-13b7b9e6e937\") " pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:47.210751 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.210724 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxc64\" (UniqueName: \"kubernetes.io/projected/c40564fb-5f5a-4698-83fe-e5d0bc2c1d58-kube-api-access-lxc64\") pod \"network-check-target-qnl77\" (UID: \"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58\") " pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:47.287653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.287612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:12:47.289915 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.289896 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 01 21:12:47.298733 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:47.298714 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 01 21:12:47.298789 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:47.298779 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:13:51.29875735 +0000 UTC m=+130.353289425 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : secret "metrics-daemon-secret" not found Jul 01 21:12:47.403872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.403838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2xqn\"" Jul 01 21:12:47.412547 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.412522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:47.500283 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.500256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gl4jg" Jul 01 21:12:47.535735 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.535704 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnl77"] Jul 01 21:12:47.540408 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:12:47.540380 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40564fb_5f5a_4698_83fe_e5d0bc2c1d58.slice/crio-6b005f35a960cdfa66c52f4a805011ec358f5812591c339f91737a74bbb77e78 WatchSource:0}: Error finding container 6b005f35a960cdfa66c52f4a805011ec358f5812591c339f91737a74bbb77e78: Status 404 returned error can't find the container with id 6b005f35a960cdfa66c52f4a805011ec358f5812591c339f91737a74bbb77e78 Jul 01 21:12:47.615749 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.615718 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gl4jg"] Jul 01 21:12:47.618664 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:12:47.618639 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55550433_0881_4a26_b1e8_13b7b9e6e937.slice/crio-5f9a1a13d881110d17de9633c1aae3cdad4655f2662b83a912889957a0fb3527 WatchSource:0}: Error finding container 5f9a1a13d881110d17de9633c1aae3cdad4655f2662b83a912889957a0fb3527: Status 404 returned error can't find the container with id 5f9a1a13d881110d17de9633c1aae3cdad4655f2662b83a912889957a0fb3527 Jul 01 21:12:47.780543 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.780455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gl4jg" event={"ID":"55550433-0881-4a26-b1e8-13b7b9e6e937","Type":"ContainerStarted","Data":"5f9a1a13d881110d17de9633c1aae3cdad4655f2662b83a912889957a0fb3527"} Jul 01 21:12:47.781489 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:47.781465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnl77" event={"ID":"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58","Type":"ContainerStarted","Data":"6b005f35a960cdfa66c52f4a805011ec358f5812591c339f91737a74bbb77e78"} Jul 01 21:12:48.093250 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:48.092914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:12:48.093250 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:48.093098 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:12:48.093250 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:12:48.093167 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:13:20.093145404 +0000 UTC m=+99.147677469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:12:50.792459 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:50.792417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnl77" event={"ID":"c40564fb-5f5a-4698-83fe-e5d0bc2c1d58","Type":"ContainerStarted","Data":"40b7303f8cc2a4325f836ef9fb2d7e8ab2d729f7bbb357b809a020cb8a1d2034"} Jul 01 21:12:50.792907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:50.792529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:12:50.806833 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:50.806745 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qnl77" podStartSLOduration=66.797263746 podStartE2EDuration="1m9.806727782s" totalImagesPullingTime="3.009464036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:50.806447254 +0000 UTC m=+69.860979335" watchObservedRunningTime="2026-07-01 21:12:50.806727782 +0000 UTC m=+69.861259866" Jul 01 21:12:52.798806 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:52.798729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gl4jg" event={"ID":"55550433-0881-4a26-b1e8-13b7b9e6e937","Type":"ContainerStarted","Data":"3c51f1676f064f5ce1b59b57db4118194cbd664d78643e962c085053e500b5da"} Jul 01 21:12:52.814471 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:12:52.814423 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gl4jg" podStartSLOduration=66.897145855 podStartE2EDuration="1m11.814407314s" totalImagesPullingTime="4.917261459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:12:52.81383639 +0000 UTC m=+71.868368472" watchObservedRunningTime="2026-07-01 21:12:52.814407314 +0000 UTC m=+71.868939432" Jul 01 21:13:14.230714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.230575 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9"] Jul 01 21:13:14.233414 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.233392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.233676 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.233648 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp"] Jul 01 21:13:14.235395 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.235378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.235484 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.235424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 01 21:13:14.235928 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.235913 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 01 21:13:14.236153 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.236131 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 01 21:13:14.236292 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.236170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-65vlk\"" Jul 01 21:13:14.236476 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.236460 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 01 21:13:14.237898 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.237883 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 01 21:13:14.237982 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.237903 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 01 21:13:14.238220 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.238202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 01 21:13:14.238268 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.238169 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 01 21:13:14.246644 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.246622 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9"] Jul 01 21:13:14.249606 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.249587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp"] Jul 01 21:13:14.276699 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm5s\" (UniqueName: \"kubernetes.io/projected/7b4777ac-8b6e-4e23-953a-08fe99cb601c-kube-api-access-8sm5s\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.276855 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.276855 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.276855 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e846b26-d85d-497a-8855-70d65f151103-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.276855 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.276998 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.276998 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b4777ac-8b6e-4e23-953a-08fe99cb601c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.276998 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.276929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b97s\" (UniqueName: \"kubernetes.io/projected/8e846b26-d85d-497a-8855-70d65f151103-kube-api-access-2b97s\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378086 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b4777ac-8b6e-4e23-953a-08fe99cb601c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.378216 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b97s\" (UniqueName: \"kubernetes.io/projected/8e846b26-d85d-497a-8855-70d65f151103-kube-api-access-2b97s\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378216 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm5s\" (UniqueName: \"kubernetes.io/projected/7b4777ac-8b6e-4e23-953a-08fe99cb601c-kube-api-access-8sm5s\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.378216 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e846b26-d85d-497a-8855-70d65f151103-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.378374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.379025 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.378999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e846b26-d85d-497a-8855-70d65f151103-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.380738 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.380715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.380842 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.380768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-ca\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.380908 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.380893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b4777ac-8b6e-4e23-953a-08fe99cb601c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.381127 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.381111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.381163 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.381120 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e846b26-d85d-497a-8855-70d65f151103-hub\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.385993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.385972 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm5s\" (UniqueName: \"kubernetes.io/projected/7b4777ac-8b6e-4e23-953a-08fe99cb601c-kube-api-access-8sm5s\") pod \"managed-serviceaccount-addon-agent-547db99c6b-gccv9\" (UID: \"7b4777ac-8b6e-4e23-953a-08fe99cb601c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.386662 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.386641 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b97s\" (UniqueName: \"kubernetes.io/projected/8e846b26-d85d-497a-8855-70d65f151103-kube-api-access-2b97s\") pod \"cluster-proxy-proxy-agent-64b79c6cbc-vp2fp\" (UID: \"8e846b26-d85d-497a-8855-70d65f151103\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.551849 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.551752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" Jul 01 21:13:14.557253 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.557235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:13:14.682923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.682894 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp"] Jul 01 21:13:14.685733 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:13:14.685707 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e846b26_d85d_497a_8855_70d65f151103.slice/crio-cb82a4a25b96baeb8d232bdbed886623f9380831b459edcc71ac7540c7ce1779 WatchSource:0}: Error finding container cb82a4a25b96baeb8d232bdbed886623f9380831b459edcc71ac7540c7ce1779: Status 404 returned error can't find the container with id cb82a4a25b96baeb8d232bdbed886623f9380831b459edcc71ac7540c7ce1779 Jul 01 21:13:14.695405 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.695382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9"] Jul 01 21:13:14.698332 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:13:14.698309 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b4777ac_8b6e_4e23_953a_08fe99cb601c.slice/crio-e084880ec9b0eb189ff96a3f371967b086c59aae3f4e204394102e5b569b5dee WatchSource:0}: Error finding container e084880ec9b0eb189ff96a3f371967b086c59aae3f4e204394102e5b569b5dee: Status 404 returned error can't find the container with id e084880ec9b0eb189ff96a3f371967b086c59aae3f4e204394102e5b569b5dee Jul 01 21:13:14.852478 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.852382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" event={"ID":"7b4777ac-8b6e-4e23-953a-08fe99cb601c","Type":"ContainerStarted","Data":"e084880ec9b0eb189ff96a3f371967b086c59aae3f4e204394102e5b569b5dee"} Jul 01 21:13:14.853399 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:14.853380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerStarted","Data":"cb82a4a25b96baeb8d232bdbed886623f9380831b459edcc71ac7540c7ce1779"} Jul 01 21:13:18.867785 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.867742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerStarted","Data":"9fb4eb5d4d4e1f5ae6f8162310bd3b034874e7a0a2a3aea001fe44e2afa5b5b3"} Jul 01 21:13:18.868131 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.867797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerStarted","Data":"ffff6e557f46fa963f23c8834fb4ffa5fbda17936d30efc4fc57efcc5a033f62"} Jul 01 21:13:18.868131 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.867818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerStarted","Data":"ad56403b2f5deaaffc727afcebba2b3e0b368648c2cc321bd7ae79b81274ed62"} Jul 01 21:13:18.869622 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.869594 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" event={"ID":"7b4777ac-8b6e-4e23-953a-08fe99cb601c","Type":"ContainerStarted","Data":"94b8a55ceaffeab46deadb57f60c10ec6fcad04def0262d7338ca0364458b9c1"} Jul 01 21:13:18.885675 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.885635 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" podStartSLOduration=0.879135562 podStartE2EDuration="4.885623389s" totalImagesPullingTime="4.006487827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:13:18.884766705 +0000 UTC m=+97.939298787" watchObservedRunningTime="2026-07-01 21:13:18.885623389 +0000 UTC m=+97.940155470" Jul 01 21:13:18.900364 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:18.900324 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" podStartSLOduration=0.916637811 podStartE2EDuration="4.900308468s" totalImagesPullingTime="3.983670657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:13:18.90012999 +0000 UTC m=+97.954662072" watchObservedRunningTime="2026-07-01 21:13:18.900308468 +0000 UTC m=+97.954840552" Jul 01 21:13:20.122397 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:20.122360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:13:20.122850 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:13:20.122510 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:13:20.122850 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:13:20.122573 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:14:24.122555925 +0000 UTC m=+163.177087985 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:13:21.797704 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:21.797667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qnl77" Jul 01 21:13:51.333627 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:13:51.333584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:13:51.334128 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:13:51.333731 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 01 21:13:51.334128 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:13:51.333805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:15:53.333786145 +0000 UTC m=+252.388318218 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : secret "metrics-daemon-secret" not found Jul 01 21:14:19.239427 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:14:19.239385 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wncw6" podUID="b5557a0e-5758-4f66-951b-8b7d55673f07" Jul 01 21:14:19.506654 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:14:19.506570 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:14:20.040412 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:14:20.040383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wncw6" Jul 01 21:14:24.153383 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:14:24.153346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:14:24.153769 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:14:24.153460 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 01 21:14:24.153769 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:14:24.153509 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls podName:b5557a0e-5758-4f66-951b-8b7d55673f07 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:26.153496671 +0000 UTC m=+285.208028731 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls") pod "dns-default-wncw6" (UID: "b5557a0e-5758-4f66-951b-8b7d55673f07") : secret "dns-default-metrics-tls" not found Jul 01 21:14:33.490879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:14:33.490790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:15:19.199237 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:15:19.199205 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b4777ac-8b6e-4e23-953a-08fe99cb601c" containerID="94b8a55ceaffeab46deadb57f60c10ec6fcad04def0262d7338ca0364458b9c1" exitCode=255 Jul 01 21:15:19.199641 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:15:19.199269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" event={"ID":"7b4777ac-8b6e-4e23-953a-08fe99cb601c","Type":"ContainerDied","Data":"94b8a55ceaffeab46deadb57f60c10ec6fcad04def0262d7338ca0364458b9c1"} Jul 01 21:15:19.199641 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:15:19.199547 2577 scope.go:117] "RemoveContainer" containerID="94b8a55ceaffeab46deadb57f60c10ec6fcad04def0262d7338ca0364458b9c1" Jul 01 21:15:20.203759 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:15:20.203726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-547db99c6b-gccv9" event={"ID":"7b4777ac-8b6e-4e23-953a-08fe99cb601c","Type":"ContainerStarted","Data":"592106d15a24d17afe957fa31cd6b90677c3594da03058d91fcffd2bd6d5fa74"} Jul 01 21:15:53.368739 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:15:53.368689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:15:53.369266 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:15:53.368847 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 01 21:15:53.369266 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:15:53.368914 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs podName:a911bcd8-9282-410b-8039-b9d043324899 nodeName:}" failed. No retries permitted until 2026-07-01 21:17:55.368896991 +0000 UTC m=+374.423429052 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs") pod "network-metrics-daemon-q79ff" (UID: "a911bcd8-9282-410b-8039-b9d043324899") : secret "metrics-daemon-secret" not found Jul 01 21:16:04.173914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.173833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9qthx"] Jul 01 21:16:04.176017 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.176000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.177939 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.177917 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 01 21:16:04.178051 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.177917 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 01 21:16:04.178115 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.177976 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 01 21:16:04.178115 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.177981 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 01 21:16:04.178239 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.177981 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dkmtq\"" Jul 01 21:16:04.192782 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.192762 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9qthx"] Jul 01 21:16:04.234336 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.234299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.234498 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.234361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec91955f-f427-4415-9149-53eb1e51f071-crio-socket\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.234498 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.234419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.234498 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.234443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec91955f-f427-4415-9149-53eb1e51f071-data-volume\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.234618 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.234490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2wg\" (UniqueName: \"kubernetes.io/projected/ec91955f-f427-4415-9149-53eb1e51f071-kube-api-access-jt2wg\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335660 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec91955f-f427-4415-9149-53eb1e51f071-data-volume\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335660 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2wg\" (UniqueName: \"kubernetes.io/projected/ec91955f-f427-4415-9149-53eb1e51f071-kube-api-access-jt2wg\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335886 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335886 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec91955f-f427-4415-9149-53eb1e51f071-crio-socket\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335886 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.335886 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.335838 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 21:16:04.335886 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.335863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ec91955f-f427-4415-9149-53eb1e51f071-crio-socket\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.336092 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.335901 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 21:16:04.336092 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.335917 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:04.835901204 +0000 UTC m=+263.890433269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : configmap "kube-rbac-proxy" not found Jul 01 21:16:04.336092 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.335960 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:04.835942232 +0000 UTC m=+263.890474309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : secret "insights-runtime-extractor-tls" not found Jul 01 21:16:04.336092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.336000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec91955f-f427-4415-9149-53eb1e51f071-data-volume\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.346914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.346894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2wg\" (UniqueName: \"kubernetes.io/projected/ec91955f-f427-4415-9149-53eb1e51f071-kube-api-access-jt2wg\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.838573 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.838541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.838784 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:04.838587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:04.838784 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.838714 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 21:16:04.838784 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.838732 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 21:16:04.838944 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.838789 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:05.838768572 +0000 UTC m=+264.893300632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : secret "insights-runtime-extractor-tls" not found Jul 01 21:16:04.838944 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:04.838808 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:05.838798782 +0000 UTC m=+264.893330842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : configmap "kube-rbac-proxy" not found Jul 01 21:16:05.844304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:05.844217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:05.844304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:05.844265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:05.844695 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:05.844345 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 21:16:05.844695 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:05.844368 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 21:16:05.844695 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:05.844391 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:07.844378506 +0000 UTC m=+266.898910566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : configmap "kube-rbac-proxy" not found Jul 01 21:16:05.844695 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:05.844449 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:07.844429648 +0000 UTC m=+266.898961722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : secret "insights-runtime-extractor-tls" not found Jul 01 21:16:06.375611 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:06.375579 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zc8cn_b8d70855-fe25-4e82-9b5f-694d5890e5be/dns-node-resolver/0.log" Jul 01 21:16:07.177577 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:07.177550 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6vwh_c9b97c52-11f1-4a3e-9d3c-f75126b58de3/node-ca/0.log" Jul 01 21:16:07.860499 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:07.860457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:07.860708 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:07.860530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:07.860708 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:07.860597 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 21:16:07.860708 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:07.860616 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 21:16:07.860708 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:07.860670 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:11.860651602 +0000 UTC m=+270.915183662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : secret "insights-runtime-extractor-tls" not found Jul 01 21:16:07.860708 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:07.860684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:11.860677504 +0000 UTC m=+270.915209564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : configmap "kube-rbac-proxy" not found Jul 01 21:16:07.989300 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:07.989272 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cvzw6_a87629f6-6a84-4c9e-9a9d-730de62f9df1/serve-healthcheck-canary/0.log" Jul 01 21:16:11.890500 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:11.890445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:11.890500 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:11.890511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:11.890955 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:11.890583 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 01 21:16:11.890955 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:11.890595 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 01 21:16:11.890955 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:11.890658 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:19.890643973 +0000 UTC m=+278.945176033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : configmap "kube-rbac-proxy" not found Jul 01 21:16:11.890955 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:11.890670 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls podName:ec91955f-f427-4415-9149-53eb1e51f071 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:19.89066468 +0000 UTC m=+278.945196740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9qthx" (UID: "ec91955f-f427-4415-9149-53eb1e51f071") : secret "insights-runtime-extractor-tls" not found Jul 01 21:16:19.952011 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:19.951968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:19.952422 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:19.952023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:19.952624 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:19.952594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ec91955f-f427-4415-9149-53eb1e51f071-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:19.954314 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:19.954294 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ec91955f-f427-4415-9149-53eb1e51f071-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9qthx\" (UID: \"ec91955f-f427-4415-9149-53eb1e51f071\") " pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:20.084541 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:20.084498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9qthx" Jul 01 21:16:20.201562 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:20.201472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9qthx"] Jul 01 21:16:20.204614 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:20.204587 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec91955f_f427_4415_9149_53eb1e51f071.slice/crio-ea139c710b6d661a1e9a73fea9ac4969e42402aebddeabf55168eb0afc78521a WatchSource:0}: Error finding container ea139c710b6d661a1e9a73fea9ac4969e42402aebddeabf55168eb0afc78521a: Status 404 returned error can't find the container with id ea139c710b6d661a1e9a73fea9ac4969e42402aebddeabf55168eb0afc78521a Jul 01 21:16:20.373039 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:20.372997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9qthx" event={"ID":"ec91955f-f427-4415-9149-53eb1e51f071","Type":"ContainerStarted","Data":"ea139c710b6d661a1e9a73fea9ac4969e42402aebddeabf55168eb0afc78521a"} Jul 01 21:16:23.041528 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:23.041480 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wncw6" podUID="b5557a0e-5758-4f66-951b-8b7d55673f07" Jul 01 21:16:23.383229 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:23.383142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:24.387452 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:24.387418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9qthx" event={"ID":"ec91955f-f427-4415-9149-53eb1e51f071","Type":"ContainerStarted","Data":"08c8c4af123da8904f104feb1d351aa1d2742ee2386a0c9c1105faa250c7a2a1"} Jul 01 21:16:25.392504 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:25.392447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9qthx" event={"ID":"ec91955f-f427-4415-9149-53eb1e51f071","Type":"ContainerStarted","Data":"1e3497cfde6f70d154ea0b60e0ff6a2484ee32b4742cccd35e0eb69647723956"} Jul 01 21:16:26.199891 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:26.199850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:26.202493 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:26.202461 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5557a0e-5758-4f66-951b-8b7d55673f07-metrics-tls\") pod \"dns-default-wncw6\" (UID: \"b5557a0e-5758-4f66-951b-8b7d55673f07\") " pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:26.386683 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:26.386651 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmdpg\"" Jul 01 21:16:26.394111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:26.394086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:27.592686 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:27.592658 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wncw6"] Jul 01 21:16:27.595838 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:27.595811 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5557a0e_5758_4f66_951b_8b7d55673f07.slice/crio-9c411669e45e3f0645193bc7f46b7f3b08d37a70b33661b7bb7c6d8eafad579d WatchSource:0}: Error finding container 9c411669e45e3f0645193bc7f46b7f3b08d37a70b33661b7bb7c6d8eafad579d: Status 404 returned error can't find the container with id 9c411669e45e3f0645193bc7f46b7f3b08d37a70b33661b7bb7c6d8eafad579d Jul 01 21:16:28.403834 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:28.403798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9qthx" event={"ID":"ec91955f-f427-4415-9149-53eb1e51f071","Type":"ContainerStarted","Data":"c352588dd0929c66ebcb8c3eb93d6721daca4ca2beaec74727aebd01fa287197"} Jul 01 21:16:28.405106 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:28.405078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wncw6" event={"ID":"b5557a0e-5758-4f66-951b-8b7d55673f07","Type":"ContainerStarted","Data":"9c411669e45e3f0645193bc7f46b7f3b08d37a70b33661b7bb7c6d8eafad579d"} Jul 01 21:16:28.420960 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:28.420906 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9qthx" podStartSLOduration=17.201983858 podStartE2EDuration="24.420889891s" totalImagesPullingTime="7.218906033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:16:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:28.419658672 +0000 UTC m=+287.474190756" watchObservedRunningTime="2026-07-01 21:16:28.420889891 +0000 UTC m=+287.475421951" Jul 01 21:16:30.412521 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:30.412489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wncw6" event={"ID":"b5557a0e-5758-4f66-951b-8b7d55673f07","Type":"ContainerStarted","Data":"6c84a69f47d5db0a03a85c279464c51c2f18b2dad4d5321a8829db6272f311f8"} Jul 01 21:16:30.413005 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:30.412537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wncw6" event={"ID":"b5557a0e-5758-4f66-951b-8b7d55673f07","Type":"ContainerStarted","Data":"b30f139e5b3c4c271a2b485884b2944c2c487f9c83799751ddb2188558bd2295"} Jul 01 21:16:30.413005 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:30.412627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:30.427757 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:30.427714 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wncw6" podStartSLOduration=252.100731726 podStartE2EDuration="4m14.427697805s" totalImagesPullingTime="2.326966079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:30.426967565 +0000 UTC m=+289.481499697" watchObservedRunningTime="2026-07-01 21:16:30.427697805 +0000 UTC m=+289.482229865" Jul 01 21:16:33.802082 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.802048 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp"] Jul 01 21:16:33.803916 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.803900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:33.805706 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.805687 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cfghr\"" Jul 01 21:16:33.805876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.805864 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 01 21:16:33.813497 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.813469 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp"] Jul 01 21:16:33.956907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:33.956867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-wb2bp\" (UID: \"cbca3372-01e5-4aa5-82cf-32fca4f20917\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:34.058319 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.058242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-wb2bp\" (UID: \"cbca3372-01e5-4aa5-82cf-32fca4f20917\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:34.058471 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:34.058445 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 01 21:16:34.058557 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:34.058544 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates podName:cbca3372-01e5-4aa5-82cf-32fca4f20917 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:34.5585193 +0000 UTC m=+293.613051363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates") pod "prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" (UID: "cbca3372-01e5-4aa5-82cf-32fca4f20917") : secret "prometheus-operator-admission-webhook-tls" not found Jul 01 21:16:34.562823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.562791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-wb2bp\" (UID: \"cbca3372-01e5-4aa5-82cf-32fca4f20917\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:34.565131 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.565102 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cbca3372-01e5-4aa5-82cf-32fca4f20917-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-wb2bp\" (UID: \"cbca3372-01e5-4aa5-82cf-32fca4f20917\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:34.712401 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.712351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:34.769075 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.769038 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:16:34.771779 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.771753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.775967 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.775940 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 01 21:16:34.776660 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.776638 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 01 21:16:34.776768 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.776707 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 01 21:16:34.776867 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.776845 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 01 21:16:34.776966 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.776911 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 01 21:16:34.777585 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.777558 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 01 21:16:34.777724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.777631 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-88hqs\"" Jul 01 21:16:34.777724 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.777688 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 01 21:16:34.780493 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.780473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 01 21:16:34.783973 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.783914 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:16:34.837224 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.837131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp"] Jul 01 21:16:34.839962 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:34.839939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbca3372_01e5_4aa5_82cf_32fca4f20917.slice/crio-39d0732bba2bc36563bb799b4c83e7f4ec3620d0514292116abb8a73f99f6504 WatchSource:0}: Error finding container 39d0732bba2bc36563bb799b4c83e7f4ec3620d0514292116abb8a73f99f6504: Status 404 returned error can't find the container with id 39d0732bba2bc36563bb799b4c83e7f4ec3620d0514292116abb8a73f99f6504 Jul 01 21:16:34.864756 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.864853 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.864853 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.864923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.864923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.864923 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.865014 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.864930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bpv\" (UniqueName: \"kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966534 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966534 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966534 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966679 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.966679 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.966581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bpv\" (UniqueName: \"kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.967239 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.967216 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.967330 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.967274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.967482 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.967458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.967781 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.967762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.968813 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.968793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.968912 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.968894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:34.974933 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:34.974915 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bpv\" (UniqueName: \"kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv\") pod \"console-775697d4b7-6k9mn\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:35.082349 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:35.082308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:35.206759 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:35.206730 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:16:35.209308 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:35.209284 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04edd805_e158_4e69_8757_5a8b85abc24e.slice/crio-0c891650a82b8ba20d9123ced5091c8bc4203813ca7ba7266ec7c11f8e141246 WatchSource:0}: Error finding container 0c891650a82b8ba20d9123ced5091c8bc4203813ca7ba7266ec7c11f8e141246: Status 404 returned error can't find the container with id 0c891650a82b8ba20d9123ced5091c8bc4203813ca7ba7266ec7c11f8e141246 Jul 01 21:16:35.428517 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:35.428485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" event={"ID":"cbca3372-01e5-4aa5-82cf-32fca4f20917","Type":"ContainerStarted","Data":"39d0732bba2bc36563bb799b4c83e7f4ec3620d0514292116abb8a73f99f6504"} Jul 01 21:16:35.429507 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:35.429484 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775697d4b7-6k9mn" event={"ID":"04edd805-e158-4e69-8757-5a8b85abc24e","Type":"ContainerStarted","Data":"0c891650a82b8ba20d9123ced5091c8bc4203813ca7ba7266ec7c11f8e141246"} Jul 01 21:16:36.492330 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:36.492281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q79ff" podUID="a911bcd8-9282-410b-8039-b9d043324899" Jul 01 21:16:37.441363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:37.441322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" event={"ID":"cbca3372-01e5-4aa5-82cf-32fca4f20917","Type":"ContainerStarted","Data":"4dcf79037f8e3c5c7d2406a03ca21e062a2016ebdc17c44c4e536be5e4bad688"} Jul 01 21:16:37.441573 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:37.441551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:37.446970 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:37.446947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" Jul 01 21:16:37.457833 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:37.457793 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-wb2bp" podStartSLOduration=2.894304361 podStartE2EDuration="4.457778887s" totalImagesPullingTime="1.563474526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:37.456782843 +0000 UTC m=+296.511314926" watchObservedRunningTime="2026-07-01 21:16:37.457778887 +0000 UTC m=+296.512310972" Jul 01 21:16:38.446034 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:38.446004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775697d4b7-6k9mn" event={"ID":"04edd805-e158-4e69-8757-5a8b85abc24e","Type":"ContainerStarted","Data":"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e"} Jul 01 21:16:38.462628 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:38.462583 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-775697d4b7-6k9mn" podStartSLOduration=1.534712194 podStartE2EDuration="4.462566905s" totalImagesPullingTime="2.927854711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:38.461644456 +0000 UTC m=+297.516176540" watchObservedRunningTime="2026-07-01 21:16:38.462566905 +0000 UTC m=+297.517098987" Jul 01 21:16:40.418157 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:40.418123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wncw6" Jul 01 21:16:41.406750 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:41.406726 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 01 21:16:45.082782 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:45.082744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:45.085060 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:45.082820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:45.087535 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:45.087517 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:45.471670 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:45.471643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:16:50.220405 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.220363 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx"] Jul 01 21:16:50.223442 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.223418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.225717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.225685 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 01 21:16:50.225868 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.225806 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 01 21:16:50.226067 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.226050 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 01 21:16:50.226417 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.226401 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 01 21:16:50.226513 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.226436 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mxncm\"" Jul 01 21:16:50.226581 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.226508 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 01 21:16:50.240504 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.240482 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx"] Jul 01 21:16:50.263969 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.263945 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sm4bn"] Jul 01 21:16:50.267069 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.267053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.270159 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.270133 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 01 21:16:50.270332 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.270319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 01 21:16:50.270402 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.270377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhdsp\"" Jul 01 21:16:50.270491 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.270475 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 01 21:16:50.275552 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-textfile\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275633 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.275633 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-sys\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275633 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-metrics-client-ca\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275729 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275729 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmffj\" (UniqueName: \"kubernetes.io/projected/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-kube-api-access-bmffj\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.275794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.275794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-root\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275882 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.275882 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275882 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqd97\" (UniqueName: \"kubernetes.io/projected/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-kube-api-access-lqd97\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.275882 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.275843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-wtmp\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376140 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.376321 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-sys\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376321 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-metrics-client-ca\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376321 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-sys\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376321 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmffj\" (UniqueName: \"kubernetes.io/projected/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-kube-api-access-bmffj\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:50.376412 2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-root\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376496 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:50.376478 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls podName:5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:50.876457346 +0000 UTC m=+309.930989420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls") pod "openshift-state-metrics-57f99c8b49-qgvzx" (UID: "5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70") : secret "openshift-state-metrics-tls" not found Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:50.376526 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqd97\" (UniqueName: \"kubernetes.io/projected/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-kube-api-access-lqd97\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-wtmp\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:16:50.376583 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls podName:8f08d2dc-39bb-42ba-bbeb-dff005d4be88 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:50.876565667 +0000 UTC m=+309.931097728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls") pod "node-exporter-sm4bn" (UID: "8f08d2dc-39bb-42ba-bbeb-dff005d4be88") : secret "node-exporter-tls" not found Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-textfile\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-wtmp\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376526 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-root\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.376794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-metrics-client-ca\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.377242 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.377242 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.376934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-textfile\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.377242 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.377124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.378929 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.378908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.379141 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.379121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.394215 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.394167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqd97\" (UniqueName: \"kubernetes.io/projected/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-kube-api-access-lqd97\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.394315 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.394265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmffj\" (UniqueName: \"kubernetes.io/projected/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-kube-api-access-bmffj\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.879331 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.879292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.879500 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.879357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:50.881714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.881688 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f08d2dc-39bb-42ba-bbeb-dff005d4be88-node-exporter-tls\") pod \"node-exporter-sm4bn\" (UID: \"8f08d2dc-39bb-42ba-bbeb-dff005d4be88\") " pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:50.881867 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:50.881714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-qgvzx\" (UID: \"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:51.133287 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.133196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" Jul 01 21:16:51.176396 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.176366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm4bn" Jul 01 21:16:51.187467 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:51.187442 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f08d2dc_39bb_42ba_bbeb_dff005d4be88.slice/crio-bfd2a479784c8cc96c4f68ff8c165f7102b50574631c843c93842eb3988713ef WatchSource:0}: Error finding container bfd2a479784c8cc96c4f68ff8c165f7102b50574631c843c93842eb3988713ef: Status 404 returned error can't find the container with id bfd2a479784c8cc96c4f68ff8c165f7102b50574631c843c93842eb3988713ef Jul 01 21:16:51.189149 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.189129 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 01 21:16:51.262032 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.261998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx"] Jul 01 21:16:51.264427 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:51.264402 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6bb4aa_8f9e_4c4b_b0d4_c6ca52791f70.slice/crio-b5ea07ea049a0ef9a8620e3e7cda39faa52616d479ac1765412262f71a101c8d WatchSource:0}: Error finding container b5ea07ea049a0ef9a8620e3e7cda39faa52616d479ac1765412262f71a101c8d: Status 404 returned error can't find the container with id b5ea07ea049a0ef9a8620e3e7cda39faa52616d479ac1765412262f71a101c8d Jul 01 21:16:51.487219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.487170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm4bn" event={"ID":"8f08d2dc-39bb-42ba-bbeb-dff005d4be88","Type":"ContainerStarted","Data":"bfd2a479784c8cc96c4f68ff8c165f7102b50574631c843c93842eb3988713ef"} Jul 01 21:16:51.488851 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.488832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" event={"ID":"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70","Type":"ContainerStarted","Data":"c08bcac3aa2661bf114ac16cde276100559960b35094273185cd0eec534fb802"} Jul 01 21:16:51.488913 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.488896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" event={"ID":"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70","Type":"ContainerStarted","Data":"dcb6cb69c350d9fe36ed74ab38a2a44f34c9363a9819d9f123a0d218e282c0fb"} Jul 01 21:16:51.488975 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.488913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" event={"ID":"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70","Type":"ContainerStarted","Data":"b5ea07ea049a0ef9a8620e3e7cda39faa52616d479ac1765412262f71a101c8d"} Jul 01 21:16:51.492275 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:51.492258 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:16:52.493241 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:52.493210 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f08d2dc-39bb-42ba-bbeb-dff005d4be88" containerID="2e034a05db54b954ccaae2c3f24e14f005c487ae3e0de743f9a225d0bc0e75d6" exitCode=0 Jul 01 21:16:52.493619 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:52.493263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm4bn" event={"ID":"8f08d2dc-39bb-42ba-bbeb-dff005d4be88","Type":"ContainerDied","Data":"2e034a05db54b954ccaae2c3f24e14f005c487ae3e0de743f9a225d0bc0e75d6"} Jul 01 21:16:53.498125 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:53.498091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" event={"ID":"5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70","Type":"ContainerStarted","Data":"76cf5498a920387d55be40597050e0caf261502b1bab73da569059f34c1118bd"} Jul 01 21:16:53.500034 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:53.500012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm4bn" event={"ID":"8f08d2dc-39bb-42ba-bbeb-dff005d4be88","Type":"ContainerStarted","Data":"521034c084dde066dc22bbbedbd1a0b7469261d6180a3f320c9cfe0d731a3537"} Jul 01 21:16:53.500152 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:53.500049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm4bn" event={"ID":"8f08d2dc-39bb-42ba-bbeb-dff005d4be88","Type":"ContainerStarted","Data":"74db78474de07ab6f69052246715b21fdd8d21c29fa8d64f7a2046d1a18c9e4c"} Jul 01 21:16:53.517237 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:53.517189 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-qgvzx" podStartSLOduration=2.036833881 podStartE2EDuration="3.517160546s" totalImagesPullingTime="1.480326665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:53.516042873 +0000 UTC m=+312.570574958" watchObservedRunningTime="2026-07-01 21:16:53.517160546 +0000 UTC m=+312.571692628" Jul 01 21:16:53.534380 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:53.534341 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sm4bn" podStartSLOduration=2.587391479 podStartE2EDuration="3.534329311s" totalImagesPullingTime="885.542687ms" totalInitContainerRuntime="61.395145ms" isStatefulPod=true podCreationTimestamp="2026-07-01 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:53.53314107 +0000 UTC m=+312.587673174" watchObservedRunningTime="2026-07-01 21:16:53.534329311 +0000 UTC m=+312.588861393" Jul 01 21:16:55.037216 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.037171 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:16:55.040306 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.040289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.052315 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.052289 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:16:55.110458 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110520 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdv4\" (UniqueName: \"kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.110651 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.110647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.211815 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.211815 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdv4\" (UniqueName: \"kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.211982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212736 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.212712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.212794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.212814 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.212887 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.212855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.214143 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.214122 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.214321 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.214306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.219973 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.219952 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdv4\" (UniqueName: \"kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4\") pod \"console-59b9b74549-84vqd\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.349381 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.349272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:16:55.477850 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.477828 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:16:55.480440 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:55.480411 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecd081c_62b9_45da_b3c1_bb27fff6e853.slice/crio-1d5dcf026d54b5723dead1ab05f63ce27943b723bb8b51b2c6b24b28bb4d05b1 WatchSource:0}: Error finding container 1d5dcf026d54b5723dead1ab05f63ce27943b723bb8b51b2c6b24b28bb4d05b1: Status 404 returned error can't find the container with id 1d5dcf026d54b5723dead1ab05f63ce27943b723bb8b51b2c6b24b28bb4d05b1 Jul 01 21:16:55.507252 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:55.507224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9b74549-84vqd" event={"ID":"becd081c-62b9-45da-b3c1-bb27fff6e853","Type":"ContainerStarted","Data":"1d5dcf026d54b5723dead1ab05f63ce27943b723bb8b51b2c6b24b28bb4d05b1"} Jul 01 21:16:56.512623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.512591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9b74549-84vqd" event={"ID":"becd081c-62b9-45da-b3c1-bb27fff6e853","Type":"ContainerStarted","Data":"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc"} Jul 01 21:16:56.527497 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.527469 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 01 21:16:56.535560 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.535519 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59b9b74549-84vqd" podStartSLOduration=1.53550437 podStartE2EDuration="1.53550437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:56.534186588 +0000 UTC m=+315.588718661" watchObservedRunningTime="2026-07-01 21:16:56.53550437 +0000 UTC m=+315.590036451" Jul 01 21:16:56.540222 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.540197 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.551158 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.551136 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 01 21:16:56.551395 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.551373 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 01 21:16:56.551824 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.551809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7rtsn8hgklr8u\"" Jul 01 21:16:56.552116 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552101 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 01 21:16:56.552193 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552165 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 01 21:16:56.552471 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 01 21:16:56.552573 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 01 21:16:56.552573 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552531 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 01 21:16:56.552699 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552597 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 01 21:16:56.552755 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552731 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 01 21:16:56.552950 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 01 21:16:56.552995 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.552937 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 01 21:16:56.553300 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.553287 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 01 21:16:56.553971 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.553957 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbmbq\"" Jul 01 21:16:56.554307 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.554294 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 01 21:16:56.554356 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.554334 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 01 21:16:56.560334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.560313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 01 21:16:56.564707 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.564678 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 01 21:16:56.622773 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622773 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrld\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-kube-api-access-gxrld\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.622993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.622940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.623426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.623379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724566 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724566 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.724873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrld\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-kube-api-access-gxrld\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.724975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725238 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725871 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725834 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.725982 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.725963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.726535 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.726196 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.727242 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.727217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.728289 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.728260 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.728384 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.728317 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.728906 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.728874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.729242 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.729220 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.729446 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.729400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.729790 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.729771 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.729884 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.729862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.730776 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.730714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.730925 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.730903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.731041 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.730960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.731433 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.731415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.731497 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.731417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.731842 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.731826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.733257 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.733170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrld\" (UniqueName: \"kubernetes.io/projected/5e79cb2e-0ecd-4dc1-9d12-9620ca46b630-kube-api-access-gxrld\") pod \"prometheus-k8s-0\" (UID: \"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:56.850371 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:56.850297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:16:57.066641 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:57.066591 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 01 21:16:57.068651 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:16:57.068626 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e79cb2e_0ecd_4dc1_9d12_9620ca46b630.slice/crio-008b804c4b35d635a341d0211abee098548997f0ceb171f03997356584a214e0 WatchSource:0}: Error finding container 008b804c4b35d635a341d0211abee098548997f0ceb171f03997356584a214e0: Status 404 returned error can't find the container with id 008b804c4b35d635a341d0211abee098548997f0ceb171f03997356584a214e0 Jul 01 21:16:57.516514 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:57.516478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"008b804c4b35d635a341d0211abee098548997f0ceb171f03997356584a214e0"} Jul 01 21:16:59.525567 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:59.525532 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e79cb2e-0ecd-4dc1-9d12-9620ca46b630" containerID="e7f061cecbcfbbdd879f68e6707be9ca713142bc366ba3384d8dd30b54503ba9" exitCode=0 Jul 01 21:16:59.526023 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:16:59.525623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerDied","Data":"e7f061cecbcfbbdd879f68e6707be9ca713142bc366ba3384d8dd30b54503ba9"} Jul 01 21:17:03.544250 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:03.544207 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"564330b116208f317d700a35b171989e6ebcf35dbeafd259c173ceadd06acf8f"} Jul 01 21:17:03.544747 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:03.544302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"f3dbab9087b930ebf5c1e22e059746e93d773e0013dedb21046e203440c857d0"} Jul 01 21:17:05.349683 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:05.349636 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:17:05.350146 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:05.349760 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:17:05.355896 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:05.355869 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:17:05.556720 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:05.556690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:17:05.601302 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:05.601212 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:17:06.558397 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.558364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"85cf0199b4d45725ca979f0a74546dc93de750d3b8e8c0206e1d4f716d12e35f"} Jul 01 21:17:06.558879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.558437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"6e1a744a778987e354ebb3c8a27b408dfe41ec49e86a59459dc6ef973831bdda"} Jul 01 21:17:06.558879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.558477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"8f313a7e0836011321bbc6b6680933ee697570b96362239a8cbc46cc8ffacb6d"} Jul 01 21:17:06.558879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.558524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e79cb2e-0ecd-4dc1-9d12-9620ca46b630","Type":"ContainerStarted","Data":"5ee0f13532a9240c760c171303294b813c509994606ed895eeaf514f3e76a4ec"} Jul 01 21:17:06.586578 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.586525 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.04202431 podStartE2EDuration="10.5865079s" totalImagesPullingTime="7.51976402s" totalInitContainerRuntime="24.71957ms" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:06.584681535 +0000 UTC m=+325.639213616" watchObservedRunningTime="2026-07-01 21:17:06.5865079 +0000 UTC m=+325.641039981" Jul 01 21:17:06.850675 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:06.850595 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:17:10.347705 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:10.347677 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cvzw6_a87629f6-6a84-4c9e-9a9d-730de62f9df1/serve-healthcheck-canary/0.log" Jul 01 21:17:14.559052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:14.558998 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" podUID="8e846b26-d85d-497a-8855-70d65f151103" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 01 21:17:24.558287 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:24.558246 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" podUID="8e846b26-d85d-497a-8855-70d65f151103" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 01 21:17:30.627958 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:30.627909 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-775697d4b7-6k9mn" podUID="04edd805-e158-4e69-8757-5a8b85abc24e" containerName="console" containerID="cri-o://75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e" gracePeriod=15 Jul 01 21:17:30.869269 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:30.869245 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:17:31.017709 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017678 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017722 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017761 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017778 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8bpv\" (UniqueName: \"kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017825 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.017895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.017851 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert\") pod \"04edd805-e158-4e69-8757-5a8b85abc24e\" (UID: \"04edd805-e158-4e69-8757-5a8b85abc24e\") " Jul 01 21:17:31.018219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.018104 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:17:31.018219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.018199 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca" (OuterVolumeSpecName: "service-ca") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:17:31.018346 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.018322 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:17:31.018346 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.018332 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config" (OuterVolumeSpecName: "console-config") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:17:31.020152 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.020124 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:17:31.020152 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.020133 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv" (OuterVolumeSpecName: "kube-api-access-q8bpv") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "kube-api-access-q8bpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:17:31.020152 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.020145 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04edd805-e158-4e69-8757-5a8b85abc24e" (UID: "04edd805-e158-4e69-8757-5a8b85abc24e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:17:31.118876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118844 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-oauth-serving-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.118876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118871 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-oauth-config\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.118876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118881 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-service-ca\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.119092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118889 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-trusted-ca-bundle\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.119092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118897 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bpv\" (UniqueName: \"kubernetes.io/projected/04edd805-e158-4e69-8757-5a8b85abc24e-kube-api-access-q8bpv\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.119092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118906 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04edd805-e158-4e69-8757-5a8b85abc24e-console-config\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.119092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.118914 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04edd805-e158-4e69-8757-5a8b85abc24e-console-serving-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:17:31.649426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.649388 2577 generic.go:356] "Generic (PLEG): container finished" podID="04edd805-e158-4e69-8757-5a8b85abc24e" containerID="75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e" exitCode=0 Jul 01 21:17:31.649811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.649455 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775697d4b7-6k9mn" Jul 01 21:17:31.649811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.649473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775697d4b7-6k9mn" event={"ID":"04edd805-e158-4e69-8757-5a8b85abc24e","Type":"ContainerDied","Data":"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e"} Jul 01 21:17:31.649811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.649552 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775697d4b7-6k9mn" event={"ID":"04edd805-e158-4e69-8757-5a8b85abc24e","Type":"ContainerDied","Data":"0c891650a82b8ba20d9123ced5091c8bc4203813ca7ba7266ec7c11f8e141246"} Jul 01 21:17:31.649811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.649573 2577 scope.go:117] "RemoveContainer" containerID="75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e" Jul 01 21:17:31.657629 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.657609 2577 scope.go:117] "RemoveContainer" containerID="75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e" Jul 01 21:17:31.657882 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:17:31.657862 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e\": container with ID starting with 75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e not found: ID does not exist" containerID="75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e" Jul 01 21:17:31.657931 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.657891 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e"} err="failed to get container status \"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e\": rpc error: code = NotFound desc = could not find container \"75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e\": container with ID starting with 75cd77e219f6f32f0afbd73fec4317fe656cabded7eec15f5ef7779d88b29f8e not found: ID does not exist" Jul 01 21:17:31.664632 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.664611 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:17:31.668403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:31.668384 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-775697d4b7-6k9mn"] Jul 01 21:17:33.494986 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:33.494915 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04edd805-e158-4e69-8757-5a8b85abc24e" path="/var/lib/kubelet/pods/04edd805-e158-4e69-8757-5a8b85abc24e/volumes" Jul 01 21:17:34.558884 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:34.558846 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" podUID="8e846b26-d85d-497a-8855-70d65f151103" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 01 21:17:34.559233 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:34.558916 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" Jul 01 21:17:34.559421 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:34.559404 2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"9fb4eb5d4d4e1f5ae6f8162310bd3b034874e7a0a2a3aea001fe44e2afa5b5b3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 01 21:17:34.559462 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:34.559443 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" podUID="8e846b26-d85d-497a-8855-70d65f151103" containerName="service-proxy" containerID="cri-o://9fb4eb5d4d4e1f5ae6f8162310bd3b034874e7a0a2a3aea001fe44e2afa5b5b3" gracePeriod=30 Jul 01 21:17:35.665415 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:35.665387 2577 generic.go:356] "Generic (PLEG): container finished" podID="8e846b26-d85d-497a-8855-70d65f151103" containerID="9fb4eb5d4d4e1f5ae6f8162310bd3b034874e7a0a2a3aea001fe44e2afa5b5b3" exitCode=2 Jul 01 21:17:35.665779 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:35.665452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerDied","Data":"9fb4eb5d4d4e1f5ae6f8162310bd3b034874e7a0a2a3aea001fe44e2afa5b5b3"} Jul 01 21:17:35.665779 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:35.665495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64b79c6cbc-vp2fp" event={"ID":"8e846b26-d85d-497a-8855-70d65f151103","Type":"ContainerStarted","Data":"5cc557e0182b8a97b6c345909a4102a7c4652a4c0a0fe3a079793e9f1aae3046"} Jul 01 21:17:55.409913 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:55.409880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:17:55.412107 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:55.412089 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a911bcd8-9282-410b-8039-b9d043324899-metrics-certs\") pod \"network-metrics-daemon-q79ff\" (UID: \"a911bcd8-9282-410b-8039-b9d043324899\") " pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:17:55.695095 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:55.695022 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xqnfx\"" Jul 01 21:17:55.703488 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:55.703466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q79ff" Jul 01 21:17:55.826291 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:55.826261 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q79ff"] Jul 01 21:17:55.829981 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:17:55.829956 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda911bcd8_9282_410b_8039_b9d043324899.slice/crio-98eb862b25ad4d3f43aeddaa5393218ed34851e4cd6e7aad0259ba054583ffc0 WatchSource:0}: Error finding container 98eb862b25ad4d3f43aeddaa5393218ed34851e4cd6e7aad0259ba054583ffc0: Status 404 returned error can't find the container with id 98eb862b25ad4d3f43aeddaa5393218ed34851e4cd6e7aad0259ba054583ffc0 Jul 01 21:17:56.738954 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:56.738923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q79ff" event={"ID":"a911bcd8-9282-410b-8039-b9d043324899","Type":"ContainerStarted","Data":"38497a78b199b651307759d3ac694727afde737a1f1d6bc52e3d6fcf07968662"} Jul 01 21:17:56.739285 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:56.738978 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q79ff" event={"ID":"a911bcd8-9282-410b-8039-b9d043324899","Type":"ContainerStarted","Data":"98eb862b25ad4d3f43aeddaa5393218ed34851e4cd6e7aad0259ba054583ffc0"} Jul 01 21:17:56.850964 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:56.850930 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:17:56.873188 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:56.873148 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:17:57.743756 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:57.743715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q79ff" event={"ID":"a911bcd8-9282-410b-8039-b9d043324899","Type":"ContainerStarted","Data":"61ef5d42e9c5eddd68b139a79d3b21b876391d911433770ffe703384776b69e8"} Jul 01 21:17:57.759424 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:57.759398 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 01 21:17:57.760063 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:17:57.760023 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q79ff" podStartSLOduration=375.958593483 podStartE2EDuration="6m16.760008163s" totalImagesPullingTime="801.41468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:57.758695004 +0000 UTC m=+376.813227090" watchObservedRunningTime="2026-07-01 21:17:57.760008163 +0000 UTC m=+376.814540245" Jul 01 21:18:35.870405 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:18:35.870371 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:19:00.893791 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:00.893751 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-59b9b74549-84vqd" podUID="becd081c-62b9-45da-b3c1-bb27fff6e853" containerName="console" containerID="cri-o://36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc" gracePeriod=15 Jul 01 21:19:01.122881 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.122855 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:19:01.209059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.208980 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209025 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwdv4\" (UniqueName: \"kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209053 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209298 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209298 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209261 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209298 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209443 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209330 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config\") pod \"becd081c-62b9-45da-b3c1-bb27fff6e853\" (UID: \"becd081c-62b9-45da-b3c1-bb27fff6e853\") " Jul 01 21:19:01.209507 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209432 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca" (OuterVolumeSpecName: "service-ca") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:19:01.209717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209628 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:19:01.209717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209676 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-service-ca\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.209717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209701 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:19:01.209717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.209718 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config" (OuterVolumeSpecName: "console-config") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:19:01.211283 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.211265 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:19:01.211349 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.211293 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:19:01.211386 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.211352 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4" (OuterVolumeSpecName: "kube-api-access-gwdv4") pod "becd081c-62b9-45da-b3c1-bb27fff6e853" (UID: "becd081c-62b9-45da-b3c1-bb27fff6e853"). InnerVolumeSpecName "kube-api-access-gwdv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:19:01.310538 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310499 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdv4\" (UniqueName: \"kubernetes.io/projected/becd081c-62b9-45da-b3c1-bb27fff6e853-kube-api-access-gwdv4\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.310538 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310531 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-oauth-config\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.310538 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310544 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-trusted-ca-bundle\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.310768 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310555 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/becd081c-62b9-45da-b3c1-bb27fff6e853-console-serving-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.310768 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310567 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-oauth-serving-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.310768 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.310580 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/becd081c-62b9-45da-b3c1-bb27fff6e853-console-config\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:19:01.965128 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.965100 2577 generic.go:356] "Generic (PLEG): container finished" podID="becd081c-62b9-45da-b3c1-bb27fff6e853" containerID="36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc" exitCode=0 Jul 01 21:19:01.965590 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.965146 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9b74549-84vqd" Jul 01 21:19:01.965590 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.965151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9b74549-84vqd" event={"ID":"becd081c-62b9-45da-b3c1-bb27fff6e853","Type":"ContainerDied","Data":"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc"} Jul 01 21:19:01.965590 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.965215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9b74549-84vqd" event={"ID":"becd081c-62b9-45da-b3c1-bb27fff6e853","Type":"ContainerDied","Data":"1d5dcf026d54b5723dead1ab05f63ce27943b723bb8b51b2c6b24b28bb4d05b1"} Jul 01 21:19:01.965590 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.965235 2577 scope.go:117] "RemoveContainer" containerID="36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc" Jul 01 21:19:01.977166 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.977143 2577 scope.go:117] "RemoveContainer" containerID="36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc" Jul 01 21:19:01.977576 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:19:01.977556 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc\": container with ID starting with 36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc not found: ID does not exist" containerID="36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc" Jul 01 21:19:01.977650 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.977583 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc"} err="failed to get container status \"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc\": rpc error: code = NotFound desc = could not find container \"36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc\": container with ID starting with 36d1b00346fb0309b14500d661b4f80e43f191a49fda2e9f3b7c44862d1b3ddc not found: ID does not exist" Jul 01 21:19:01.986918 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.986886 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:19:01.990119 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:01.990098 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59b9b74549-84vqd"] Jul 01 21:19:03.494787 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:19:03.494716 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="becd081c-62b9-45da-b3c1-bb27fff6e853" path="/var/lib/kubelet/pods/becd081c-62b9-45da-b3c1-bb27fff6e853/volumes" Jul 01 21:20:39.861221 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861128 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w"] Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861400 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04edd805-e158-4e69-8757-5a8b85abc24e" containerName="console" Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861411 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04edd805-e158-4e69-8757-5a8b85abc24e" containerName="console" Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861421 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="becd081c-62b9-45da-b3c1-bb27fff6e853" containerName="console" Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861427 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="becd081c-62b9-45da-b3c1-bb27fff6e853" containerName="console" Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861481 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04edd805-e158-4e69-8757-5a8b85abc24e" containerName="console" Jul 01 21:20:39.861725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.861491 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="becd081c-62b9-45da-b3c1-bb27fff6e853" containerName="console" Jul 01 21:20:39.863441 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.863425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.867667 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.867642 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-b94hx\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b94hx\"" type="*v1.Secret" Jul 01 21:20:39.867808 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.867643 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 01 21:20:39.867808 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.867745 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 01 21:20:39.868470 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.868441 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 01 21:20:39.868470 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.868441 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 01 21:20:39.868470 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:39.868437 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="de5798ac-b6be-4758-90a8-9c77abcd4dc3" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.938598 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.938566 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w"] Jul 01 21:20:39.969315 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.969281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-manifests\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.969315 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.969319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.969501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.969342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.969501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.969380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-tmp\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:39.969501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:39.969397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gx4\" (UniqueName: \"kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070247 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-tmp\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gx4\" (UniqueName: \"kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-manifests\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.070715 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.070699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-manifests\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.072457 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.072437 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de5798ac-b6be-4758-90a8-9c77abcd4dc3-tmp\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.698690 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.698662 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 01 21:20:40.703171 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.703136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:40.703171 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:40.703136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de5798ac-b6be-4758-90a8-9c77abcd4dc3-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:41.102465 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:41.102378 2577 projected.go:291] Couldn't get configMap opendatahub/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 01 21:20:41.305189 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.305155 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 01 21:20:41.361201 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.361104 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b94hx\"" Jul 01 21:20:41.410742 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.410718 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 01 21:20:41.412821 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:41.412804 2577 projected.go:196] Error preparing data for projected volume kube-api-access-r4gx4 for pod opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w: failed to sync configmap cache: timed out waiting for the condition Jul 01 21:20:41.412882 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:20:41.412872 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4 podName:de5798ac-b6be-4758-90a8-9c77abcd4dc3 nodeName:}" failed. No retries permitted until 2026-07-01 21:20:41.912855673 +0000 UTC m=+540.967387733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4gx4" (UniqueName: "kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4") pod "opendatahub-operator-controller-manager-85d96b6c7-fz76w" (UID: "de5798ac-b6be-4758-90a8-9c77abcd4dc3") : failed to sync configmap cache: timed out waiting for the condition Jul 01 21:20:41.454330 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.454306 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 01 21:20:41.985698 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.985659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gx4\" (UniqueName: \"kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:41.988075 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:41.988057 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gx4\" (UniqueName: \"kubernetes.io/projected/de5798ac-b6be-4758-90a8-9c77abcd4dc3-kube-api-access-r4gx4\") pod \"opendatahub-operator-controller-manager-85d96b6c7-fz76w\" (UID: \"de5798ac-b6be-4758-90a8-9c77abcd4dc3\") " pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:42.275796 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:42.275715 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b94hx\"" Jul 01 21:20:42.283915 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:42.283893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:42.424230 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:42.424202 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w"] Jul 01 21:20:42.427535 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:20:42.427508 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5798ac_b6be_4758_90a8_9c77abcd4dc3.slice/crio-777a5b03b331c9fd3a21494485951fd9116956e0180af00c0abc444d7dc13e6a WatchSource:0}: Error finding container 777a5b03b331c9fd3a21494485951fd9116956e0180af00c0abc444d7dc13e6a: Status 404 returned error can't find the container with id 777a5b03b331c9fd3a21494485951fd9116956e0180af00c0abc444d7dc13e6a Jul 01 21:20:43.318778 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:43.318744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" event={"ID":"de5798ac-b6be-4758-90a8-9c77abcd4dc3","Type":"ContainerStarted","Data":"777a5b03b331c9fd3a21494485951fd9116956e0180af00c0abc444d7dc13e6a"} Jul 01 21:20:45.328336 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:45.328306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" event={"ID":"de5798ac-b6be-4758-90a8-9c77abcd4dc3","Type":"ContainerStarted","Data":"15e5f22c48b46c1714025ca09324707733e37e450907064a2e85381068683c34"} Jul 01 21:20:47.336950 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:47.336917 2577 generic.go:356] "Generic (PLEG): container finished" podID="de5798ac-b6be-4758-90a8-9c77abcd4dc3" containerID="15e5f22c48b46c1714025ca09324707733e37e450907064a2e85381068683c34" exitCode=0 Jul 01 21:20:47.337421 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:47.336958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" event={"ID":"de5798ac-b6be-4758-90a8-9c77abcd4dc3","Type":"ContainerDied","Data":"15e5f22c48b46c1714025ca09324707733e37e450907064a2e85381068683c34"} Jul 01 21:20:48.343052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:48.343011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" event={"ID":"de5798ac-b6be-4758-90a8-9c77abcd4dc3","Type":"ContainerStarted","Data":"ec096b7fe5d3dffa6f3bb694cb0edce9666db832986453fdc7856b83bd122397"} Jul 01 21:20:48.343471 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:48.343312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:20:48.368360 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:48.368308 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" podStartSLOduration=5.10284209 podStartE2EDuration="9.368292419s" totalImagesPullingTime="3.200511354s" totalInitContainerRuntime="1.064938975s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:20:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:20:48.365064858 +0000 UTC m=+547.419596940" watchObservedRunningTime="2026-07-01 21:20:48.368292419 +0000 UTC m=+547.422824500" Jul 01 21:20:59.349950 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:20:59.349917 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85d96b6c7-fz76w" Jul 01 21:21:01.787287 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.787254 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb"] Jul 01 21:21:01.797631 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.797606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:01.801448 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.801431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 01 21:21:01.802316 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.802297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nfz4r\"" Jul 01 21:21:01.802442 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.802331 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 01 21:21:01.802442 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.802333 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 01 21:21:01.805562 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.805541 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 01 21:21:01.806927 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.806906 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb"] Jul 01 21:21:01.958277 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.958236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6x9v\" (UniqueName: \"kubernetes.io/projected/a7771f04-6862-445d-96a3-6263fb781f5a-kube-api-access-p6x9v\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:01.958277 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.958279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7771f04-6862-445d-96a3-6263fb781f5a-tmp\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:01.958522 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:01.958303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7771f04-6862-445d-96a3-6263fb781f5a-tls-certs\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.059712 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.059628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7771f04-6862-445d-96a3-6263fb781f5a-tls-certs\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.059712 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.059702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6x9v\" (UniqueName: \"kubernetes.io/projected/a7771f04-6862-445d-96a3-6263fb781f5a-kube-api-access-p6x9v\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.059891 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.059724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7771f04-6862-445d-96a3-6263fb781f5a-tmp\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.061963 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.061936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7771f04-6862-445d-96a3-6263fb781f5a-tmp\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.062063 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.062049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7771f04-6862-445d-96a3-6263fb781f5a-tls-certs\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.064381 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.064358 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-78qzx"] Jul 01 21:21:02.066530 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.066511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.069526 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.069510 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hbd88\"" Jul 01 21:21:02.069740 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.069726 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 01 21:21:02.075638 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.075618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6x9v\" (UniqueName: \"kubernetes.io/projected/a7771f04-6862-445d-96a3-6263fb781f5a-kube-api-access-p6x9v\") pod \"kube-auth-proxy-5cb45b86b-lg2zb\" (UID: \"a7771f04-6862-445d-96a3-6263fb781f5a\") " pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.083588 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.083564 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-78qzx"] Jul 01 21:21:02.108067 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.108031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" Jul 01 21:21:02.160464 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.160429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/872757d0-baca-4bfa-a2c5-9e1ea4009289-cert\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.160577 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.160476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnpz\" (UniqueName: \"kubernetes.io/projected/872757d0-baca-4bfa-a2c5-9e1ea4009289-kube-api-access-xjnpz\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.235603 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.235571 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb"] Jul 01 21:21:02.238848 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:02.238825 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7771f04_6862_445d_96a3_6263fb781f5a.slice/crio-65640bd7ff64b4a24c024326275b3719dde98649edcb171a7c4386f09ea36d3b WatchSource:0}: Error finding container 65640bd7ff64b4a24c024326275b3719dde98649edcb171a7c4386f09ea36d3b: Status 404 returned error can't find the container with id 65640bd7ff64b4a24c024326275b3719dde98649edcb171a7c4386f09ea36d3b Jul 01 21:21:02.261496 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.261472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/872757d0-baca-4bfa-a2c5-9e1ea4009289-cert\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.261613 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.261506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnpz\" (UniqueName: \"kubernetes.io/projected/872757d0-baca-4bfa-a2c5-9e1ea4009289-kube-api-access-xjnpz\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.263872 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.263849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/872757d0-baca-4bfa-a2c5-9e1ea4009289-cert\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.269006 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.268982 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnpz\" (UniqueName: \"kubernetes.io/projected/872757d0-baca-4bfa-a2c5-9e1ea4009289-kube-api-access-xjnpz\") pod \"odh-model-controller-7998c67dc4-78qzx\" (UID: \"872757d0-baca-4bfa-a2c5-9e1ea4009289\") " pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.388734 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.388650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:02.392898 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.392870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" event={"ID":"a7771f04-6862-445d-96a3-6263fb781f5a","Type":"ContainerStarted","Data":"65640bd7ff64b4a24c024326275b3719dde98649edcb171a7c4386f09ea36d3b"} Jul 01 21:21:02.514316 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:02.514282 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-78qzx"] Jul 01 21:21:02.517284 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:02.517257 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872757d0_baca_4bfa_a2c5_9e1ea4009289.slice/crio-5c7472d23d502e5e5e2b55f1da25cf8fb19df6cb93dc5dda6d690c4982918f48 WatchSource:0}: Error finding container 5c7472d23d502e5e5e2b55f1da25cf8fb19df6cb93dc5dda6d690c4982918f48: Status 404 returned error can't find the container with id 5c7472d23d502e5e5e2b55f1da25cf8fb19df6cb93dc5dda6d690c4982918f48 Jul 01 21:21:03.222889 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.222846 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qqx9d"] Jul 01 21:21:03.225635 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.225612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.228785 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.228764 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 01 21:21:03.229090 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.229039 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-x8gcg\"" Jul 01 21:21:03.248544 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.248514 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qqx9d"] Jul 01 21:21:03.370252 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.370205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.370439 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.370291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lc2\" (UniqueName: \"kubernetes.io/projected/1fae2f73-a3e4-4ff3-8419-fd931ce61125-kube-api-access-55lc2\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.398194 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.398146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" event={"ID":"872757d0-baca-4bfa-a2c5-9e1ea4009289","Type":"ContainerStarted","Data":"5c7472d23d502e5e5e2b55f1da25cf8fb19df6cb93dc5dda6d690c4982918f48"} Jul 01 21:21:03.418425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.418396 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:03.420698 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.420678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.422955 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.422928 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 01 21:21:03.423061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.422966 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 01 21:21:03.423061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.422928 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 01 21:21:03.423361 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.423219 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 01 21:21:03.423361 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.423256 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qm75x\"" Jul 01 21:21:03.455687 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.455658 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:03.470743 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.470720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.470869 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.470783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lc2\" (UniqueName: \"kubernetes.io/projected/1fae2f73-a3e4-4ff3-8419-fd931ce61125-kube-api-access-55lc2\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.470908 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:21:03.470876 2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 01 21:21:03.470950 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:21:03.470940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs podName:1fae2f73-a3e4-4ff3-8419-fd931ce61125 nodeName:}" failed. No retries permitted until 2026-07-01 21:21:03.970920706 +0000 UTC m=+563.025452773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs") pod "model-serving-api-54d89bd5fc-qqx9d" (UID: "1fae2f73-a3e4-4ff3-8419-fd931ce61125") : secret "model-serving-api-tls" not found Jul 01 21:21:03.487690 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.487616 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lc2\" (UniqueName: \"kubernetes.io/projected/1fae2f73-a3e4-4ff3-8419-fd931ce61125-kube-api-access-55lc2\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.571934 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.571739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.571934 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.571804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.571934 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.571840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.571934 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.571876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.572311 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.571970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.572311 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.572012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4lj\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.572311 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.572131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673028 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.672981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673028 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673304 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4lj\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673517 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.673806 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.673779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.675529 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.675500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.675653 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.675593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.675753 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.675735 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.675904 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.675886 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.680621 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.680601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.681241 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.681153 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4lj\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj\") pod \"istiod-openshift-gateway-7574459656-ffr9f\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.731049 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.731019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:03.880487 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.880446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:03.883672 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:03.883643 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda851d325_a0ed_490d_8316_5bb02476572b.slice/crio-4270af808487d9c9092f78fb2f2a4fd67097627409c7fb9bdd883f018598e2a9 WatchSource:0}: Error finding container 4270af808487d9c9092f78fb2f2a4fd67097627409c7fb9bdd883f018598e2a9: Status 404 returned error can't find the container with id 4270af808487d9c9092f78fb2f2a4fd67097627409c7fb9bdd883f018598e2a9 Jul 01 21:21:03.976767 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.976726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:03.980044 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:03.980010 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fae2f73-a3e4-4ff3-8419-fd931ce61125-tls-certs\") pod \"model-serving-api-54d89bd5fc-qqx9d\" (UID: \"1fae2f73-a3e4-4ff3-8419-fd931ce61125\") " pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:04.139276 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:04.139192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:04.304808 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:04.304773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qqx9d"] Jul 01 21:21:04.308909 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:04.308879 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fae2f73_a3e4_4ff3_8419_fd931ce61125.slice/crio-7ab1d6ebd7746dc9ea64a2fdbe04985c4962d7bccd64a0248a8bb12d73ddf7ed WatchSource:0}: Error finding container 7ab1d6ebd7746dc9ea64a2fdbe04985c4962d7bccd64a0248a8bb12d73ddf7ed: Status 404 returned error can't find the container with id 7ab1d6ebd7746dc9ea64a2fdbe04985c4962d7bccd64a0248a8bb12d73ddf7ed Jul 01 21:21:04.405395 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:04.405349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" event={"ID":"1fae2f73-a3e4-4ff3-8419-fd931ce61125","Type":"ContainerStarted","Data":"7ab1d6ebd7746dc9ea64a2fdbe04985c4962d7bccd64a0248a8bb12d73ddf7ed"} Jul 01 21:21:04.406935 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:04.406892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" event={"ID":"a851d325-a0ed-490d-8316-5bb02476572b","Type":"ContainerStarted","Data":"4270af808487d9c9092f78fb2f2a4fd67097627409c7fb9bdd883f018598e2a9"} Jul 01 21:21:08.017889 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.017849 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:21:08.018195 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.017930 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:21:08.427961 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.427922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" event={"ID":"a7771f04-6862-445d-96a3-6263fb781f5a","Type":"ContainerStarted","Data":"fa7c43323c48fe61592d19ff14d7ba99ef1755275ca10fc29c08889e66c9bb34"} Jul 01 21:21:08.429486 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.429461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" event={"ID":"872757d0-baca-4bfa-a2c5-9e1ea4009289","Type":"ContainerStarted","Data":"36ec50275c17efe861177af87f1ba3fe0f659022a840c0e8865926d7d5740546"} Jul 01 21:21:08.429601 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.429512 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:08.431098 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.431064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" event={"ID":"a851d325-a0ed-490d-8316-5bb02476572b","Type":"ContainerStarted","Data":"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373"} Jul 01 21:21:08.431212 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.431108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:08.432522 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.432503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" event={"ID":"1fae2f73-a3e4-4ff3-8419-fd931ce61125","Type":"ContainerStarted","Data":"24502256566d93728c7d039a6893defaf55df95f38b75cd65e2cdb00acd89415"} Jul 01 21:21:08.432620 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.432604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:08.444088 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.444050 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb45b86b-lg2zb" podStartSLOduration=1.731269041 podStartE2EDuration="7.444036498s" totalImagesPullingTime="5.712767457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:08.442685303 +0000 UTC m=+567.497217384" watchObservedRunningTime="2026-07-01 21:21:08.444036498 +0000 UTC m=+567.498568579" Jul 01 21:21:08.460969 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.460918 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" podStartSLOduration=1.8115017340000001 podStartE2EDuration="5.460895555s" totalImagesPullingTime="3.649393821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:08.459622121 +0000 UTC m=+567.514154201" watchObservedRunningTime="2026-07-01 21:21:08.460895555 +0000 UTC m=+567.515427628" Jul 01 21:21:08.476099 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.476049 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" podStartSLOduration=1.041294099 podStartE2EDuration="6.476030569s" totalImagesPullingTime="5.43473647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:08.475278229 +0000 UTC m=+567.529810324" watchObservedRunningTime="2026-07-01 21:21:08.476030569 +0000 UTC m=+567.530562650" Jul 01 21:21:08.502323 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:08.502280 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" podStartSLOduration=1.371011168 podStartE2EDuration="5.502264972s" totalImagesPullingTime="4.131253804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:08.500931612 +0000 UTC m=+567.555463695" watchObservedRunningTime="2026-07-01 21:21:08.502264972 +0000 UTC m=+567.556797055" Jul 01 21:21:09.443596 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.443566 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-ffr9f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 01 21:21:09.444018 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.443620 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" podUID="a851d325-a0ed-490d-8316-5bb02476572b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 21:21:09.647801 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.647763 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc"] Jul 01 21:21:09.651098 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.651076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.653124 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.653104 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jtsdz\"" Jul 01 21:21:09.671288 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.671257 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc"] Jul 01 21:21:09.729910 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.729820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.729910 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.729895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.729935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.729964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c965a9-780d-4f35-8711-c7909a819ffa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.730006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.730039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c965a9-780d-4f35-8711-c7909a819ffa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.730068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730309 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.730105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7hl\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-kube-api-access-6t7hl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.730309 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.730199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.830852 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7hl\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-kube-api-access-6t7hl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831007 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.830985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c965a9-780d-4f35-8711-c7909a819ffa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831296 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831296 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c965a9-780d-4f35-8711-c7909a819ffa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831296 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831431 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831431 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831380 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831603 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831705 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831639 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.831939 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.831917 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c965a9-780d-4f35-8711-c7909a819ffa-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.833217 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.833197 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c965a9-780d-4f35-8711-c7909a819ffa-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.833560 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.833543 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c965a9-780d-4f35-8711-c7909a819ffa-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.845483 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.845462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.852358 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.852320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7hl\" (UniqueName: \"kubernetes.io/projected/60c965a9-780d-4f35-8711-c7909a819ffa-kube-api-access-6t7hl\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ndwdc\" (UID: \"60c965a9-780d-4f35-8711-c7909a819ffa\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:09.962030 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:09.962000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:10.097165 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:10.097134 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc"] Jul 01 21:21:10.098383 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:10.098358 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c965a9_780d_4f35_8711_c7909a819ffa.slice/crio-3913d8ac0d63284e646bf9dcf00da8e91d84635c4743a8e8a7efb0dd04a57f6b WatchSource:0}: Error finding container 3913d8ac0d63284e646bf9dcf00da8e91d84635c4743a8e8a7efb0dd04a57f6b: Status 404 returned error can't find the container with id 3913d8ac0d63284e646bf9dcf00da8e91d84635c4743a8e8a7efb0dd04a57f6b Jul 01 21:21:10.446557 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:10.446517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" event={"ID":"60c965a9-780d-4f35-8711-c7909a819ffa","Type":"ContainerStarted","Data":"3913d8ac0d63284e646bf9dcf00da8e91d84635c4743a8e8a7efb0dd04a57f6b"} Jul 01 21:21:12.201161 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.201121 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkckm"] Jul 01 21:21:12.208465 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.208443 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj"] Jul 01 21:21:12.208609 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.208573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.211675 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.211652 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 01 21:21:12.212226 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.212211 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-brlgc\"" Jul 01 21:21:12.212937 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.212919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.215595 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.215580 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jrrqz\"" Jul 01 21:21:12.215988 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.215967 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 01 21:21:12.239851 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.239817 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkckm"] Jul 01 21:21:12.241462 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.241434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj"] Jul 01 21:21:12.250146 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.250116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3658e83-440b-48c1-ba75-3236bc6fd760-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.250286 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.250161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9k4\" (UniqueName: \"kubernetes.io/projected/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-kube-api-access-cl9k4\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.250356 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.250330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97t9\" (UniqueName: \"kubernetes.io/projected/f3658e83-440b-48c1-ba75-3236bc6fd760-kube-api-access-x97t9\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.250465 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.250359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.351426 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.351381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3658e83-440b-48c1-ba75-3236bc6fd760-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.351615 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.351438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9k4\" (UniqueName: \"kubernetes.io/projected/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-kube-api-access-cl9k4\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.351615 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.351533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97t9\" (UniqueName: \"kubernetes.io/projected/f3658e83-440b-48c1-ba75-3236bc6fd760-kube-api-access-x97t9\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.351615 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.351570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.351818 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:21:12.351671 2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 01 21:21:12.351818 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:21:12.351733 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert podName:2a6a5dca-d629-4060-a7bf-7c44dbdb0bba nodeName:}" failed. No retries permitted until 2026-07-01 21:21:12.851713256 +0000 UTC m=+571.906245330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert") pod "kserve-controller-manager-d6f4b65b8-pkckm" (UID: "2a6a5dca-d629-4060-a7bf-7c44dbdb0bba") : secret "kserve-webhook-server-cert" not found Jul 01 21:21:12.354546 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.354518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3658e83-440b-48c1-ba75-3236bc6fd760-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.360732 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.360705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97t9\" (UniqueName: \"kubernetes.io/projected/f3658e83-440b-48c1-ba75-3236bc6fd760-kube-api-access-x97t9\") pod \"llmisvc-controller-manager-777c6bd7d5-lnrxj\" (UID: \"f3658e83-440b-48c1-ba75-3236bc6fd760\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.360988 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.360963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9k4\" (UniqueName: \"kubernetes.io/projected/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-kube-api-access-cl9k4\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.443520 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.443489 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:12.527777 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.527695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:12.761716 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.761679 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:21:12.761823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.761752 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:21:12.761823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.761790 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:21:12.855896 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.855858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.858562 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.858528 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a6a5dca-d629-4060-a7bf-7c44dbdb0bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkckm\" (UID: \"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:12.902000 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:12.901968 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3658e83_440b_48c1_ba75_3236bc6fd760.slice/crio-9baab5aa4b5ddce39a20ce469189dc5cf81292d7eb9da4db8fb023f7e1942212 WatchSource:0}: Error finding container 9baab5aa4b5ddce39a20ce469189dc5cf81292d7eb9da4db8fb023f7e1942212: Status 404 returned error can't find the container with id 9baab5aa4b5ddce39a20ce469189dc5cf81292d7eb9da4db8fb023f7e1942212 Jul 01 21:21:12.903835 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:12.903811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj"] Jul 01 21:21:13.121089 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.121004 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:13.245618 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.245566 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkckm"] Jul 01 21:21:13.248014 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:21:13.247981 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6a5dca_d629_4060_a7bf_7c44dbdb0bba.slice/crio-5e33c26bf6700bb5e1822b61435dc8ef52c134eac0de25615c40bd87192f32ec WatchSource:0}: Error finding container 5e33c26bf6700bb5e1822b61435dc8ef52c134eac0de25615c40bd87192f32ec: Status 404 returned error can't find the container with id 5e33c26bf6700bb5e1822b61435dc8ef52c134eac0de25615c40bd87192f32ec Jul 01 21:21:13.462537 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.462499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" event={"ID":"60c965a9-780d-4f35-8711-c7909a819ffa","Type":"ContainerStarted","Data":"9876c2186e75e2c138b7186193eb855d026cb644a5710614885883cf62184f06"} Jul 01 21:21:13.463587 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.463560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" event={"ID":"f3658e83-440b-48c1-ba75-3236bc6fd760","Type":"ContainerStarted","Data":"9baab5aa4b5ddce39a20ce469189dc5cf81292d7eb9da4db8fb023f7e1942212"} Jul 01 21:21:13.464533 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.464515 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" event={"ID":"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba","Type":"ContainerStarted","Data":"5e33c26bf6700bb5e1822b61435dc8ef52c134eac0de25615c40bd87192f32ec"} Jul 01 21:21:13.519683 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.519633 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" podStartSLOduration=1.8586585370000002 podStartE2EDuration="4.519614724s" totalImagesPullingTime="2.660956187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:13.515667338 +0000 UTC m=+572.570199420" watchObservedRunningTime="2026-07-01 21:21:13.519614724 +0000 UTC m=+572.574146866" Jul 01 21:21:13.789867 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.789781 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:13.790170 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.790146 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" podUID="a851d325-a0ed-490d-8316-5bb02476572b" containerName="discovery" containerID="cri-o://220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373" gracePeriod=30 Jul 01 21:21:13.962992 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.962960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:13.968444 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:13.968419 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:14.063356 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.063333 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:14.167582 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167753 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167753 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167753 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167714 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167753 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167959 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.167959 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.167826 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k4lj\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj\") pod \"a851d325-a0ed-490d-8316-5bb02476572b\" (UID: \"a851d325-a0ed-490d-8316-5bb02476572b\") " Jul 01 21:21:14.168059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.168016 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:21:14.170545 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.170516 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:21:14.171666 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.171617 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs" (OuterVolumeSpecName: "local-certs") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 21:21:14.171666 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.171629 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:21:14.171666 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.171641 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts" (OuterVolumeSpecName: "cacerts") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:21:14.171914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.171865 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj" (OuterVolumeSpecName: "kube-api-access-8k4lj") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "kube-api-access-8k4lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:21:14.171914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.171893 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a851d325-a0ed-490d-8316-5bb02476572b" (UID: "a851d325-a0ed-490d-8316-5bb02476572b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:21:14.268788 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268748 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-istio-token\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.268788 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268785 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4lj\" (UniqueName: \"kubernetes.io/projected/a851d325-a0ed-490d-8316-5bb02476572b-kube-api-access-8k4lj\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.269329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268800 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-ca-configmap\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.269329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268813 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-cacerts\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.269329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268826 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851d325-a0ed-490d-8316-5bb02476572b-local-certs\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.269329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268838 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-csr-dns-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.269329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.268850 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a851d325-a0ed-490d-8316-5bb02476572b-istio-kubeconfig\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:21:14.471763 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.471720 2577 generic.go:356] "Generic (PLEG): container finished" podID="a851d325-a0ed-490d-8316-5bb02476572b" containerID="220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373" exitCode=0 Jul 01 21:21:14.471991 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.471956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" event={"ID":"a851d325-a0ed-490d-8316-5bb02476572b","Type":"ContainerDied","Data":"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373"} Jul 01 21:21:14.472092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.472023 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" Jul 01 21:21:14.472092 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.472041 2577 scope.go:117] "RemoveContainer" containerID="220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373" Jul 01 21:21:14.472229 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.472026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f" event={"ID":"a851d325-a0ed-490d-8316-5bb02476572b","Type":"ContainerDied","Data":"4270af808487d9c9092f78fb2f2a4fd67097627409c7fb9bdd883f018598e2a9"} Jul 01 21:21:14.472348 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.472310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:14.473635 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.473610 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ndwdc" Jul 01 21:21:14.483474 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.483454 2577 scope.go:117] "RemoveContainer" containerID="220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373" Jul 01 21:21:14.483834 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:21:14.483770 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373\": container with ID starting with 220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373 not found: ID does not exist" containerID="220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373" Jul 01 21:21:14.483834 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.483807 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373"} err="failed to get container status \"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373\": rpc error: code = NotFound desc = could not find container \"220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373\": container with ID starting with 220c311ddeb665b6baa69b597a9c1346c3dc4d92be787dd3992538add7a68373 not found: ID does not exist" Jul 01 21:21:14.521483 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.521449 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:14.530665 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:14.530633 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ffr9f"] Jul 01 21:21:15.497269 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:15.497235 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a851d325-a0ed-490d-8316-5bb02476572b" path="/var/lib/kubelet/pods/a851d325-a0ed-490d-8316-5bb02476572b/volumes" Jul 01 21:21:16.483685 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.483652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" event={"ID":"2a6a5dca-d629-4060-a7bf-7c44dbdb0bba","Type":"ContainerStarted","Data":"77a5f91808e9f574b953686755310cbbf2ec61b4a7663028b6c7188b29454a56"} Jul 01 21:21:16.483818 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.483748 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:21:16.485049 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.485024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" event={"ID":"f3658e83-440b-48c1-ba75-3236bc6fd760","Type":"ContainerStarted","Data":"c895aa81cfa786e4f8aa49360f5487150a472da04ae36363692a55becc9167ed"} Jul 01 21:21:16.485245 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.485229 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:16.499381 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.499343 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" podStartSLOduration=1.417135399 podStartE2EDuration="4.499329653s" totalImagesPullingTime="3.082194254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:16.49805614 +0000 UTC m=+575.552588217" watchObservedRunningTime="2026-07-01 21:21:16.499329653 +0000 UTC m=+575.553861734" Jul 01 21:21:16.517129 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:16.517085 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" podStartSLOduration=1.084585612 podStartE2EDuration="4.517071304s" totalImagesPullingTime="3.432485692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:16.516494378 +0000 UTC m=+575.571026460" watchObservedRunningTime="2026-07-01 21:21:16.517071304 +0000 UTC m=+575.571603386" Jul 01 21:21:19.444298 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:19.444264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-78qzx" Jul 01 21:21:19.447010 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:19.446988 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qqx9d" Jul 01 21:21:47.495536 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:47.495508 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lnrxj" Jul 01 21:21:47.497001 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:21:47.495583 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkckm" Jul 01 21:22:02.712747 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.712668 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:02.713196 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.712996 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a851d325-a0ed-490d-8316-5bb02476572b" containerName="discovery" Jul 01 21:22:02.713196 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.713007 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a851d325-a0ed-490d-8316-5bb02476572b" containerName="discovery" Jul 01 21:22:02.713196 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.713068 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a851d325-a0ed-490d-8316-5bb02476572b" containerName="discovery" Jul 01 21:22:02.714991 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.714975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:02.716806 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.716786 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 01 21:22:02.716933 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.716788 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 01 21:22:02.716933 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.716840 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sjjzd\"" Jul 01 21:22:02.724786 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.724765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:02.770098 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.770066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkssg\" (UniqueName: \"kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg\") pod \"kuadrant-operator-catalog-hb6wf\" (UID: \"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894\") " pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:02.870879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.870842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkssg\" (UniqueName: \"kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg\") pod \"kuadrant-operator-catalog-hb6wf\" (UID: \"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894\") " pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:02.878677 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:02.878648 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkssg\" (UniqueName: \"kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg\") pod \"kuadrant-operator-catalog-hb6wf\" (UID: \"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894\") " pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:03.025659 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.025579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:03.084399 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.084370 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:03.156472 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.156447 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:03.158853 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:03.158825 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f5a2ad_a7ea_4d9f_b857_8e4535b9a894.slice/crio-67478b5e45962db31facd89cf813ab5cc1eae14df07c47361ef7b466b05d00c1 WatchSource:0}: Error finding container 67478b5e45962db31facd89cf813ab5cc1eae14df07c47361ef7b466b05d00c1: Status 404 returned error can't find the container with id 67478b5e45962db31facd89cf813ab5cc1eae14df07c47361ef7b466b05d00c1 Jul 01 21:22:03.160343 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.160322 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 01 21:22:03.290773 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.290697 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6d64x"] Jul 01 21:22:03.293545 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.293528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:03.303069 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.303045 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6d64x"] Jul 01 21:22:03.374794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.374765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j75m\" (UniqueName: \"kubernetes.io/projected/366a100e-7550-48fc-b9b3-e6924c176960-kube-api-access-7j75m\") pod \"kuadrant-operator-catalog-6d64x\" (UID: \"366a100e-7550-48fc-b9b3-e6924c176960\") " pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:03.475944 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.475881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j75m\" (UniqueName: \"kubernetes.io/projected/366a100e-7550-48fc-b9b3-e6924c176960-kube-api-access-7j75m\") pod \"kuadrant-operator-catalog-6d64x\" (UID: \"366a100e-7550-48fc-b9b3-e6924c176960\") " pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:03.483847 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.483820 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j75m\" (UniqueName: \"kubernetes.io/projected/366a100e-7550-48fc-b9b3-e6924c176960-kube-api-access-7j75m\") pod \"kuadrant-operator-catalog-6d64x\" (UID: \"366a100e-7550-48fc-b9b3-e6924c176960\") " pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:03.604072 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.603995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:03.676099 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.676063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" event={"ID":"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894","Type":"ContainerStarted","Data":"67478b5e45962db31facd89cf813ab5cc1eae14df07c47361ef7b466b05d00c1"} Jul 01 21:22:03.731123 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:03.731093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6d64x"] Jul 01 21:22:03.733863 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:03.733842 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366a100e_7550_48fc_b9b3_e6924c176960.slice/crio-45e19be7e03ad36d13d09c6067290855b802564e873bc3aa9abe289f972df4d8 WatchSource:0}: Error finding container 45e19be7e03ad36d13d09c6067290855b802564e873bc3aa9abe289f972df4d8: Status 404 returned error can't find the container with id 45e19be7e03ad36d13d09c6067290855b802564e873bc3aa9abe289f972df4d8 Jul 01 21:22:04.689349 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:04.689299 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" event={"ID":"366a100e-7550-48fc-b9b3-e6924c176960","Type":"ContainerStarted","Data":"45e19be7e03ad36d13d09c6067290855b802564e873bc3aa9abe289f972df4d8"} Jul 01 21:22:05.695119 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.695025 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" event={"ID":"366a100e-7550-48fc-b9b3-e6924c176960","Type":"ContainerStarted","Data":"6b5615f2f86a910f0507479fa34d78521ca98df0e7d53e44695f42604719cf65"} Jul 01 21:22:05.696576 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.696548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" event={"ID":"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894","Type":"ContainerStarted","Data":"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117"} Jul 01 21:22:05.696679 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.696582 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" podUID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" containerName="registry-server" containerID="cri-o://41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117" gracePeriod=2 Jul 01 21:22:05.709194 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.709126 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" podStartSLOduration=0.996033608 podStartE2EDuration="2.70910666s" totalImagesPullingTime="1.713073052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:05.708103273 +0000 UTC m=+624.762635346" watchObservedRunningTime="2026-07-01 21:22:05.70910666 +0000 UTC m=+624.763638744" Jul 01 21:22:05.722419 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.722378 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" podStartSLOduration=1.437620261 podStartE2EDuration="3.722364058s" totalImagesPullingTime="2.284743797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:05.7208675 +0000 UTC m=+624.775399583" watchObservedRunningTime="2026-07-01 21:22:05.722364058 +0000 UTC m=+624.776896145" Jul 01 21:22:05.941061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.941038 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:05.998474 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:05.998397 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkssg\" (UniqueName: \"kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg\") pod \"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894\" (UID: \"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894\") " Jul 01 21:22:06.000527 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.000501 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg" (OuterVolumeSpecName: "kube-api-access-kkssg") pod "83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" (UID: "83f5a2ad-a7ea-4d9f-b857-8e4535b9a894"). InnerVolumeSpecName "kube-api-access-kkssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:22:06.098895 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.098849 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkssg\" (UniqueName: \"kubernetes.io/projected/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894-kube-api-access-kkssg\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:22:06.701236 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.701197 2577 generic.go:356] "Generic (PLEG): container finished" podID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" containerID="41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117" exitCode=0 Jul 01 21:22:06.701236 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.701228 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" Jul 01 21:22:06.701662 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.701270 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" event={"ID":"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894","Type":"ContainerDied","Data":"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117"} Jul 01 21:22:06.701662 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.701317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hb6wf" event={"ID":"83f5a2ad-a7ea-4d9f-b857-8e4535b9a894","Type":"ContainerDied","Data":"67478b5e45962db31facd89cf813ab5cc1eae14df07c47361ef7b466b05d00c1"} Jul 01 21:22:06.701662 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.701339 2577 scope.go:117] "RemoveContainer" containerID="41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117" Jul 01 21:22:06.710369 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.710350 2577 scope.go:117] "RemoveContainer" containerID="41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117" Jul 01 21:22:06.710624 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:06.710608 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117\": container with ID starting with 41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117 not found: ID does not exist" containerID="41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117" Jul 01 21:22:06.710683 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.710637 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117"} err="failed to get container status \"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117\": rpc error: code = NotFound desc = could not find container \"41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117\": container with ID starting with 41e00c0a11b5971463700bde8a1a52b644402ddcfc42ea2197d6f4bd3e19c117 not found: ID does not exist" Jul 01 21:22:06.720592 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.720537 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:06.726542 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:06.726524 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hb6wf"] Jul 01 21:22:07.496635 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:07.496601 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" path="/var/lib/kubelet/pods/83f5a2ad-a7ea-4d9f-b857-8e4535b9a894/volumes" Jul 01 21:22:13.604674 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:13.604638 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:13.605206 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:13.604705 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:13.626953 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:13.626926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:13.754120 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:13.754095 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6d64x" Jul 01 21:22:32.175776 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.175745 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6"] Jul 01 21:22:32.176142 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.176069 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" containerName="registry-server" Jul 01 21:22:32.176142 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.176080 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" containerName="registry-server" Jul 01 21:22:32.176142 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.176132 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f5a2ad-a7ea-4d9f-b857-8e4535b9a894" containerName="registry-server" Jul 01 21:22:32.179599 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.179574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:32.181821 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.181802 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z4st2\"" Jul 01 21:22:32.189492 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.189469 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6"] Jul 01 21:22:32.324820 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.324785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxx9p\" (UniqueName: \"kubernetes.io/projected/a909f286-59b4-47fa-9631-104db00cdd78-kube-api-access-kxx9p\") pod \"authorino-operator-75cc8d5cb7-ll2w6\" (UID: \"a909f286-59b4-47fa-9631-104db00cdd78\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:32.425185 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.425136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxx9p\" (UniqueName: \"kubernetes.io/projected/a909f286-59b4-47fa-9631-104db00cdd78-kube-api-access-kxx9p\") pod \"authorino-operator-75cc8d5cb7-ll2w6\" (UID: \"a909f286-59b4-47fa-9631-104db00cdd78\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:32.436531 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.436473 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxx9p\" (UniqueName: \"kubernetes.io/projected/a909f286-59b4-47fa-9631-104db00cdd78-kube-api-access-kxx9p\") pod \"authorino-operator-75cc8d5cb7-ll2w6\" (UID: \"a909f286-59b4-47fa-9631-104db00cdd78\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:32.489329 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.489290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:32.617563 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.617530 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6"] Jul 01 21:22:32.620326 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:32.620298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda909f286_59b4_47fa_9631_104db00cdd78.slice/crio-302d11421a8db684e4c645e35d24109151e4f5e2593e56bf0fca1d57750fec9b WatchSource:0}: Error finding container 302d11421a8db684e4c645e35d24109151e4f5e2593e56bf0fca1d57750fec9b: Status 404 returned error can't find the container with id 302d11421a8db684e4c645e35d24109151e4f5e2593e56bf0fca1d57750fec9b Jul 01 21:22:32.812357 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:32.812275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" event={"ID":"a909f286-59b4-47fa-9631-104db00cdd78","Type":"ContainerStarted","Data":"302d11421a8db684e4c645e35d24109151e4f5e2593e56bf0fca1d57750fec9b"} Jul 01 21:22:35.407833 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.407801 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8"] Jul 01 21:22:35.410916 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.410898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:35.413494 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.413470 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dvpmg\"" Jul 01 21:22:35.422315 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.422293 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8"] Jul 01 21:22:35.553526 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.553490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89z5\" (UniqueName: \"kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5\") pod \"limitador-operator-controller-manager-859d7b78bd-82cl8\" (UID: \"7fdb6454-91d2-431e-ae9a-27922a2fa93d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:35.654831 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.654793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89z5\" (UniqueName: \"kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5\") pod \"limitador-operator-controller-manager-859d7b78bd-82cl8\" (UID: \"7fdb6454-91d2-431e-ae9a-27922a2fa93d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:35.668605 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.668532 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89z5\" (UniqueName: \"kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5\") pod \"limitador-operator-controller-manager-859d7b78bd-82cl8\" (UID: \"7fdb6454-91d2-431e-ae9a-27922a2fa93d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:35.720508 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.720476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:35.829950 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.829915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" event={"ID":"a909f286-59b4-47fa-9631-104db00cdd78","Type":"ContainerStarted","Data":"f86d0216fd58ef9cee0e1435ad74af1fdd39f694ce0a5250516f84c72297a2a1"} Jul 01 21:22:35.830101 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.830046 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:35.847297 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.847249 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" podStartSLOduration=1.595736646 podStartE2EDuration="3.847232922s" totalImagesPullingTime="2.251496276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:35.846467578 +0000 UTC m=+654.900999662" watchObservedRunningTime="2026-07-01 21:22:35.847232922 +0000 UTC m=+654.901765003" Jul 01 21:22:35.851512 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:35.851488 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8"] Jul 01 21:22:35.854066 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:35.854044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdb6454_91d2_431e_ae9a_27922a2fa93d.slice/crio-71d5f9bcd055c8029bafa0d62e0f202a34868c23e24b86c9ee8d19846553e062 WatchSource:0}: Error finding container 71d5f9bcd055c8029bafa0d62e0f202a34868c23e24b86c9ee8d19846553e062: Status 404 returned error can't find the container with id 71d5f9bcd055c8029bafa0d62e0f202a34868c23e24b86c9ee8d19846553e062 Jul 01 21:22:36.838014 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:36.837976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" event={"ID":"7fdb6454-91d2-431e-ae9a-27922a2fa93d","Type":"ContainerStarted","Data":"71d5f9bcd055c8029bafa0d62e0f202a34868c23e24b86c9ee8d19846553e062"} Jul 01 21:22:37.844283 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:37.844192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" event={"ID":"7fdb6454-91d2-431e-ae9a-27922a2fa93d","Type":"ContainerStarted","Data":"66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6"} Jul 01 21:22:37.844643 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:37.844310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:37.862739 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:37.862691 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" podStartSLOduration=1.303108667 podStartE2EDuration="2.862677822s" totalImagesPullingTime="1.559569155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:37.861622375 +0000 UTC m=+656.916154459" watchObservedRunningTime="2026-07-01 21:22:37.862677822 +0000 UTC m=+656.917209905" Jul 01 21:22:40.210373 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.210343 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74"] Jul 01 21:22:40.217710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.217694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.219761 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.219743 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2gkct\"" Jul 01 21:22:40.229011 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.227267 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74"] Jul 01 21:22:40.295700 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.295661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.295869 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.295724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmbt\" (UniqueName: \"kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.396722 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.396687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.396878 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.396738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmbt\" (UniqueName: \"kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.397083 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.397062 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.407876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.407849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmbt\" (UniqueName: \"kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.530073 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.529969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:40.659888 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.659858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74"] Jul 01 21:22:40.661550 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:40.661522 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7295755_ab8f_4186_8f92_7457963f41ed.slice/crio-4db60cf0e41987d9839601447ca7cf8529bd74fb536bce17fa49b29918bca95b WatchSource:0}: Error finding container 4db60cf0e41987d9839601447ca7cf8529bd74fb536bce17fa49b29918bca95b: Status 404 returned error can't find the container with id 4db60cf0e41987d9839601447ca7cf8529bd74fb536bce17fa49b29918bca95b Jul 01 21:22:40.858881 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:40.858784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" event={"ID":"d7295755-ab8f-4186-8f92-7457963f41ed","Type":"ContainerStarted","Data":"4db60cf0e41987d9839601447ca7cf8529bd74fb536bce17fa49b29918bca95b"} Jul 01 21:22:46.840888 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:46.840857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ll2w6" Jul 01 21:22:46.888823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:46.888786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" event={"ID":"d7295755-ab8f-4186-8f92-7457963f41ed","Type":"ContainerStarted","Data":"3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425"} Jul 01 21:22:46.889003 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:46.888896 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:46.908785 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:46.908739 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" podStartSLOduration=0.904903836 podStartE2EDuration="6.908721662s" totalImagesPullingTime="6.003817826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:46.908541146 +0000 UTC m=+665.963073229" watchObservedRunningTime="2026-07-01 21:22:46.908721662 +0000 UTC m=+665.963253743" Jul 01 21:22:48.851697 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:48.851664 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:57.896377 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:57.896342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.029106 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.029070 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz"] Jul 01 21:22:59.032618 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.032599 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.048375 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.048352 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz"] Jul 01 21:22:59.063917 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.063885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b482g\" (UniqueName: \"kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.064068 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.063988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.164667 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.164629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.164836 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.164686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b482g\" (UniqueName: \"kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.165108 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.165081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.174393 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.174350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b482g\" (UniqueName: \"kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g\") pod \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.343057 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.342967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.473612 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.473583 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz"] Jul 01 21:22:59.476245 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:22:59.476217 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a586e9_bec1_4b32_ad8c_7a959210eced.slice/crio-a97b976fdc3e67f2b861b169889beae5ccdbb3bf88335c20f35f732f83df9b54 WatchSource:0}: Error finding container a97b976fdc3e67f2b861b169889beae5ccdbb3bf88335c20f35f732f83df9b54: Status 404 returned error can't find the container with id a97b976fdc3e67f2b861b169889beae5ccdbb3bf88335c20f35f732f83df9b54 Jul 01 21:22:59.557010 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.556981 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz"] Jul 01 21:22:59.564952 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.564924 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz"] Jul 01 21:22:59.574486 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.574463 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74"] Jul 01 21:22:59.574710 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.574688 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" containerName="manager" containerID="cri-o://3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425" gracePeriod=2 Jul 01 21:22:59.586407 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.586387 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74"] Jul 01 21:22:59.599892 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.599794 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d7295755-ab8f-4186-8f92-7457963f41ed\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-gqx74\": pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.602456 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602437 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:22:59.602769 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602756 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" containerName="manager" Jul 01 21:22:59.602811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602771 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" containerName="manager" Jul 01 21:22:59.602811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602780 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" containerName="manager" Jul 01 21:22:59.602811 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602785 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" containerName="manager" Jul 01 21:22:59.602939 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602844 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" containerName="manager" Jul 01 21:22:59.602939 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.602854 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" containerName="manager" Jul 01 21:22:59.605695 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.605676 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8"] Jul 01 21:22:59.605794 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.605779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.605874 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.605854 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" containerName="manager" containerID="cri-o://66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6" gracePeriod=2 Jul 01 21:22:59.622091 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.621226 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:22:59.627210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.626943 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8"] Jul 01 21:22:59.634035 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.632730 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7fdb6454-91d2-431e-ae9a-27922a2fa93d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-82cl8\": pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.635837 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.635817 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj"] Jul 01 21:22:59.636150 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.636138 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" containerName="manager" Jul 01 21:22:59.636227 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.636151 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" containerName="manager" Jul 01 21:22:59.636270 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.636236 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" containerName="manager" Jul 01 21:22:59.641002 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.640986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:22:59.651485 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.651452 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj"] Jul 01 21:22:59.654504 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.654465 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.656032 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.655997 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.669143 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.669119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpmq\" (UniqueName: \"kubernetes.io/projected/051b7308-0c2f-4085-9d55-b15e1ce84296-kube-api-access-6wpmq\") pod \"limitador-operator-controller-manager-859d7b78bd-vwfvj\" (UID: \"051b7308-0c2f-4085-9d55-b15e1ce84296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:22:59.669294 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.669154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226k9\" (UniqueName: \"kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.669294 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.669196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.677827 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.677793 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.770565 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.770521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpmq\" (UniqueName: \"kubernetes.io/projected/051b7308-0c2f-4085-9d55-b15e1ce84296-kube-api-access-6wpmq\") pod \"limitador-operator-controller-manager-859d7b78bd-vwfvj\" (UID: \"051b7308-0c2f-4085-9d55-b15e1ce84296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:22:59.770686 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.770601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226k9\" (UniqueName: \"kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.770686 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.770648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.771130 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.771075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.779042 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.779023 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226k9\" (UniqueName: \"kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xr7rf\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.779130 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.779106 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpmq\" (UniqueName: \"kubernetes.io/projected/051b7308-0c2f-4085-9d55-b15e1ce84296-kube-api-access-6wpmq\") pod \"limitador-operator-controller-manager-859d7b78bd-vwfvj\" (UID: \"051b7308-0c2f-4085-9d55-b15e1ce84296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:22:59.831825 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.831677 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.833596 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.833570 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.834952 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.834932 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.838421 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.838402 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.839738 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.839719 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.840765 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.840746 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.871357 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.871301 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s89z5\" (UniqueName: \"kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5\") pod \"7fdb6454-91d2-431e-ae9a-27922a2fa93d\" (UID: \"7fdb6454-91d2-431e-ae9a-27922a2fa93d\") " Jul 01 21:22:59.871357 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.871334 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume\") pod \"d7295755-ab8f-4186-8f92-7457963f41ed\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " Jul 01 21:22:59.871357 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.871355 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxmbt\" (UniqueName: \"kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt\") pod \"d7295755-ab8f-4186-8f92-7457963f41ed\" (UID: \"d7295755-ab8f-4186-8f92-7457963f41ed\") " Jul 01 21:22:59.871834 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.871810 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d7295755-ab8f-4186-8f92-7457963f41ed" (UID: "d7295755-ab8f-4186-8f92-7457963f41ed"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 21:22:59.873388 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.873362 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt" (OuterVolumeSpecName: "kube-api-access-rxmbt") pod "d7295755-ab8f-4186-8f92-7457963f41ed" (UID: "d7295755-ab8f-4186-8f92-7457963f41ed"). InnerVolumeSpecName "kube-api-access-rxmbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:22:59.873474 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.873388 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5" (OuterVolumeSpecName: "kube-api-access-s89z5") pod "7fdb6454-91d2-431e-ae9a-27922a2fa93d" (UID: "7fdb6454-91d2-431e-ae9a-27922a2fa93d"). InnerVolumeSpecName "kube-api-access-s89z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:22:59.948810 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.948770 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" containerID="66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6" exitCode=0 Jul 01 21:22:59.948980 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.948816 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.948980 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.948878 2577 scope.go:117] "RemoveContainer" containerID="66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6" Jul 01 21:22:59.950261 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.950240 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7295755-ab8f-4186-8f92-7457963f41ed" containerID="3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425" exitCode=0 Jul 01 21:22:59.950362 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.950289 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.951454 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.951426 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.952485 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.952467 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" container="manager" Jul 01 21:22:59.953323 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.953294 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.955103 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.955078 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.956608 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.956581 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.957812 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.957797 2577 scope.go:117] "RemoveContainer" containerID="66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6" Jul 01 21:22:59.958030 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.958007 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.958109 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.958088 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6\": container with ID starting with 66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6 not found: ID does not exist" containerID="66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6" Jul 01 21:22:59.958163 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.958121 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6"} err="failed to get container status \"66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6\": rpc error: code = NotFound desc = could not find container \"66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6\": container with ID starting with 66cfd572ed36121c5aa01be524f2c204c3a425f645e082a25a9ceba1278f86f6 not found: ID does not exist" Jul 01 21:22:59.958163 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.958144 2577 scope.go:117] "RemoveContainer" containerID="3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425" Jul 01 21:22:59.962555 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.962534 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.964403 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.964379 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.965906 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.965884 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.966852 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.966837 2577 scope.go:117] "RemoveContainer" containerID="3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425" Jul 01 21:22:59.967111 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.967093 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425\": container with ID starting with 3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425 not found: ID does not exist" containerID="3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425" Jul 01 21:22:59.967190 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.967118 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425"} err="failed to get container status \"3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425\": rpc error: code = NotFound desc = could not find container \"3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425\": container with ID starting with 3d5b33eba8e86389dcb66bb24facdc779c7189df9e187be7bbb4f234d6f2e425 not found: ID does not exist" Jul 01 21:22:59.967450 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.967423 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:22:59.968983 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.968964 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:22:59.970450 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:22:59.970423 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:22:59.972984 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.972960 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89z5\" (UniqueName: \"kubernetes.io/projected/7fdb6454-91d2-431e-ae9a-27922a2fa93d-kube-api-access-s89z5\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:22:59.973059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.972992 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7295755-ab8f-4186-8f92-7457963f41ed-extensions-socket-volume\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:22:59.973059 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.973007 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxmbt\" (UniqueName: \"kubernetes.io/projected/d7295755-ab8f-4186-8f92-7457963f41ed-kube-api-access-rxmbt\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:22:59.986141 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.986120 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:22:59.991715 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:22:59.991697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:23:00.130244 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.130135 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:23:00.132865 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:00.132836 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9613836_4cee_42dc_bb9f_edf19553d62d.slice/crio-ba179180cb29e984a4b5d89db631411b02c88310193a9d787fffc5e411af7db1 WatchSource:0}: Error finding container ba179180cb29e984a4b5d89db631411b02c88310193a9d787fffc5e411af7db1: Status 404 returned error can't find the container with id ba179180cb29e984a4b5d89db631411b02c88310193a9d787fffc5e411af7db1 Jul 01 21:23:00.150018 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.149907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj"] Jul 01 21:23:00.153328 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:00.153307 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051b7308_0c2f_4085_9d55_b15e1ce84296.slice/crio-1fc56c93029faa93fe54262f9141010be64f12b84dd58b0495fc3815f9dc30da WatchSource:0}: Error finding container 1fc56c93029faa93fe54262f9141010be64f12b84dd58b0495fc3815f9dc30da: Status 404 returned error can't find the container with id 1fc56c93029faa93fe54262f9141010be64f12b84dd58b0495fc3815f9dc30da Jul 01 21:23:00.960196 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.960135 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" containerName="manager" containerID="cri-o://43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0" gracePeriod=2 Jul 01 21:23:00.961724 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:00.961695 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:23:00.962061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.961728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" event={"ID":"b9613836-4cee-42dc-bb9f-edf19553d62d","Type":"ContainerStarted","Data":"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50"} Jul 01 21:23:00.962153 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.962101 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:23:00.962153 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.962128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" event={"ID":"b9613836-4cee-42dc-bb9f-edf19553d62d","Type":"ContainerStarted","Data":"ba179180cb29e984a4b5d89db631411b02c88310193a9d787fffc5e411af7db1"} Jul 01 21:23:00.963319 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:00.963278 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:00.963319 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.963305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" event={"ID":"051b7308-0c2f-4085-9d55-b15e1ce84296","Type":"ContainerStarted","Data":"d283e0e3316de50b5039febd81caa9300972ddb5a9ed3b3a00309283751691ce"} Jul 01 21:23:00.963460 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.963346 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" event={"ID":"051b7308-0c2f-4085-9d55-b15e1ce84296","Type":"ContainerStarted","Data":"1fc56c93029faa93fe54262f9141010be64f12b84dd58b0495fc3815f9dc30da"} Jul 01 21:23:00.963460 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.963380 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:23:00.964535 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:00.964516 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:23:00.982802 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.982760 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" podStartSLOduration=1.9827446530000001 podStartE2EDuration="1.982744653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:00.980704317 +0000 UTC m=+680.035236399" watchObservedRunningTime="2026-07-01 21:23:00.982744653 +0000 UTC m=+680.037276736" Jul 01 21:23:00.997424 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:00.997379 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" podStartSLOduration=1.99736637 podStartE2EDuration="1.99736637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:00.996865346 +0000 UTC m=+680.051397427" watchObservedRunningTime="2026-07-01 21:23:00.99736637 +0000 UTC m=+680.051898451" Jul 01 21:23:00.998393 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:00.998361 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:23:00.999674 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:00.999652 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:23:01.000745 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.000728 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.208449 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.208424 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.210328 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.210268 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:23:01.211977 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.211955 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:23:01.213400 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.213380 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.283622 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.283589 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume\") pod \"80a586e9-bec1-4b32-ad8c-7a959210eced\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " Jul 01 21:23:01.283816 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.283667 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b482g\" (UniqueName: \"kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g\") pod \"80a586e9-bec1-4b32-ad8c-7a959210eced\" (UID: \"80a586e9-bec1-4b32-ad8c-7a959210eced\") " Jul 01 21:23:01.283883 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.283838 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "80a586e9-bec1-4b32-ad8c-7a959210eced" (UID: "80a586e9-bec1-4b32-ad8c-7a959210eced"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 21:23:01.285708 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.285682 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g" (OuterVolumeSpecName: "kube-api-access-b482g") pod "80a586e9-bec1-4b32-ad8c-7a959210eced" (UID: "80a586e9-bec1-4b32-ad8c-7a959210eced"). InnerVolumeSpecName "kube-api-access-b482g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:23:01.384438 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.384405 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b482g\" (UniqueName: \"kubernetes.io/projected/80a586e9-bec1-4b32-ad8c-7a959210eced-kube-api-access-b482g\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:01.384438 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.384433 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80a586e9-bec1-4b32-ad8c-7a959210eced-extensions-socket-volume\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:01.494565 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.494472 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-82cl8\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-82cl8" Jul 01 21:23:01.495747 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.495709 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.496916 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.496892 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fdb6454-91d2-431e-ae9a-27922a2fa93d" path="/var/lib/kubelet/pods/7fdb6454-91d2-431e-ae9a-27922a2fa93d/volumes" Jul 01 21:23:01.497009 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.496940 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqx74\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqx74" Jul 01 21:23:01.497281 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.497267 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" path="/var/lib/kubelet/pods/80a586e9-bec1-4b32-ad8c-7a959210eced/volumes" Jul 01 21:23:01.497572 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.497560 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7295755-ab8f-4186-8f92-7457963f41ed" path="/var/lib/kubelet/pods/d7295755-ab8f-4186-8f92-7457963f41ed/volumes" Jul 01 21:23:01.969454 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.969423 2577 generic.go:356] "Generic (PLEG): container finished" podID="80a586e9-bec1-4b32-ad8c-7a959210eced" containerID="43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0" exitCode=2 Jul 01 21:23:01.969644 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.969470 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.969644 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.969508 2577 scope.go:117] "RemoveContainer" containerID="43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0" Jul 01 21:23:01.973481 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.973450 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-gh5bz\" is forbidden: User \"system:node:ip-10-0-140-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-112.ec2.internal' and this object" podUID="80a586e9-bec1-4b32-ad8c-7a959210eced" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gh5bz" Jul 01 21:23:01.977663 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.977641 2577 scope.go:117] "RemoveContainer" containerID="43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0" Jul 01 21:23:01.977903 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:01.977887 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0\": container with ID starting with 43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0 not found: ID does not exist" containerID="43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0" Jul 01 21:23:01.977964 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:01.977913 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0"} err="failed to get container status \"43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0\": rpc error: code = NotFound desc = could not find container \"43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0\": container with ID starting with 43f7f74605a16ec1705528d400c950310eefd7714d07471f7f5bc593b4371cb0 not found: ID does not exist" Jul 01 21:23:11.971650 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:11.971621 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vwfvj" Jul 01 21:23:11.972027 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:11.971692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:23:16.283252 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.283220 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:23:16.283650 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.283395 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" podUID="b9613836-4cee-42dc-bb9f-edf19553d62d" containerName="manager" containerID="cri-o://3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50" gracePeriod=10 Jul 01 21:23:16.481951 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.481907 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8"] Jul 01 21:23:16.486424 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.486406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.505583 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.505556 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8"] Jul 01 21:23:16.609044 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.608932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.609137 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.609074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kx4\" (UniqueName: \"kubernetes.io/projected/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-kube-api-access-96kx4\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.627797 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.627774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:23:16.709962 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.709931 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume\") pod \"b9613836-4cee-42dc-bb9f-edf19553d62d\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " Jul 01 21:23:16.710136 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.709977 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-226k9\" (UniqueName: \"kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9\") pod \"b9613836-4cee-42dc-bb9f-edf19553d62d\" (UID: \"b9613836-4cee-42dc-bb9f-edf19553d62d\") " Jul 01 21:23:16.710136 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.710121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kx4\" (UniqueName: \"kubernetes.io/projected/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-kube-api-access-96kx4\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.710226 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.710154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.710330 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.710307 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9613836-4cee-42dc-bb9f-edf19553d62d" (UID: "b9613836-4cee-42dc-bb9f-edf19553d62d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 01 21:23:16.710520 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.710505 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.711947 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.711927 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9" (OuterVolumeSpecName: "kube-api-access-226k9") pod "b9613836-4cee-42dc-bb9f-edf19553d62d" (UID: "b9613836-4cee-42dc-bb9f-edf19553d62d"). InnerVolumeSpecName "kube-api-access-226k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:23:16.722585 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.722562 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kx4\" (UniqueName: \"kubernetes.io/projected/3e0792f4-c321-423f-8eb2-4c7d68eb9f77-kube-api-access-96kx4\") pod \"kuadrant-operator-controller-manager-86878c769f-mqlw8\" (UID: \"3e0792f4-c321-423f-8eb2-4c7d68eb9f77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.810943 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.810862 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9613836-4cee-42dc-bb9f-edf19553d62d-extensions-socket-volume\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:16.810943 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.810889 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226k9\" (UniqueName: \"kubernetes.io/projected/b9613836-4cee-42dc-bb9f-edf19553d62d-kube-api-access-226k9\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:16.816936 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.816914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:16.950224 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:16.950190 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8"] Jul 01 21:23:16.953708 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:16.953684 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0792f4_c321_423f_8eb2_4c7d68eb9f77.slice/crio-b96574f5b42a2abdc20cf8df3498da1315a368f5b0179915d041883ddd87a710 WatchSource:0}: Error finding container b96574f5b42a2abdc20cf8df3498da1315a368f5b0179915d041883ddd87a710: Status 404 returned error can't find the container with id b96574f5b42a2abdc20cf8df3498da1315a368f5b0179915d041883ddd87a710 Jul 01 21:23:17.042350 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.042319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" event={"ID":"3e0792f4-c321-423f-8eb2-4c7d68eb9f77","Type":"ContainerStarted","Data":"803ddce39ef56cd08da04a5111e642e05380650ffd96ec2c1ae7750c71d0fa88"} Jul 01 21:23:17.042487 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.042375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" event={"ID":"3e0792f4-c321-423f-8eb2-4c7d68eb9f77","Type":"ContainerStarted","Data":"b96574f5b42a2abdc20cf8df3498da1315a368f5b0179915d041883ddd87a710"} Jul 01 21:23:17.042487 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.042402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:17.043687 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.043659 2577 generic.go:356] "Generic (PLEG): container finished" podID="b9613836-4cee-42dc-bb9f-edf19553d62d" containerID="3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50" exitCode=0 Jul 01 21:23:17.043807 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.043691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" event={"ID":"b9613836-4cee-42dc-bb9f-edf19553d62d","Type":"ContainerDied","Data":"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50"} Jul 01 21:23:17.043807 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.043713 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" Jul 01 21:23:17.043807 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.043747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf" event={"ID":"b9613836-4cee-42dc-bb9f-edf19553d62d","Type":"ContainerDied","Data":"ba179180cb29e984a4b5d89db631411b02c88310193a9d787fffc5e411af7db1"} Jul 01 21:23:17.043807 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.043779 2577 scope.go:117] "RemoveContainer" containerID="3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50" Jul 01 21:23:17.052378 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.052361 2577 scope.go:117] "RemoveContainer" containerID="3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50" Jul 01 21:23:17.052656 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:17.052636 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50\": container with ID starting with 3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50 not found: ID does not exist" containerID="3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50" Jul 01 21:23:17.052742 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.052661 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50"} err="failed to get container status \"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50\": rpc error: code = NotFound desc = could not find container \"3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50\": container with ID starting with 3691081b8236d96258ce50497e65f0d584d65c9e0a7d5d7c96f2c11515832e50 not found: ID does not exist" Jul 01 21:23:17.064105 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.064001 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" podStartSLOduration=1.063984004 podStartE2EDuration="1.063984004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:17.061763462 +0000 UTC m=+696.116295548" watchObservedRunningTime="2026-07-01 21:23:17.063984004 +0000 UTC m=+696.118516086" Jul 01 21:23:17.076755 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.076720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:23:17.078851 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.078830 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xr7rf"] Jul 01 21:23:17.496654 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:17.496626 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9613836-4cee-42dc-bb9f-edf19553d62d" path="/var/lib/kubelet/pods/b9613836-4cee-42dc-bb9f-edf19553d62d/volumes" Jul 01 21:23:23.725210 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.725150 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69dcc88757-8mlvr"] Jul 01 21:23:23.725879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.725519 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9613836-4cee-42dc-bb9f-edf19553d62d" containerName="manager" Jul 01 21:23:23.725879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.725532 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9613836-4cee-42dc-bb9f-edf19553d62d" containerName="manager" Jul 01 21:23:23.725879 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.725585 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9613836-4cee-42dc-bb9f-edf19553d62d" containerName="manager" Jul 01 21:23:23.728158 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.728132 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.730615 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.730596 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 01 21:23:23.730805 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.730792 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 01 21:23:23.731378 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731357 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 01 21:23:23.731378 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731380 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 01 21:23:23.731570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 01 21:23:23.731570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731465 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-88hqs\"" Jul 01 21:23:23.731648 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 01 21:23:23.731648 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.731616 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 01 21:23:23.737371 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.737350 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 01 21:23:23.740066 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.740049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dcc88757-8mlvr"] Jul 01 21:23:23.875252 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-trusted-ca-bundle\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875252 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-service-ca\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-oauth-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-oauth-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.875501 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.875397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsjr\" (UniqueName: \"kubernetes.io/projected/92a41dfb-e6fa-409c-ae78-feb7f5423642-kube-api-access-nvsjr\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976469 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-oauth-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976469 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-oauth-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976658 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976658 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976658 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsjr\" (UniqueName: \"kubernetes.io/projected/92a41dfb-e6fa-409c-ae78-feb7f5423642-kube-api-access-nvsjr\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976829 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-trusted-ca-bundle\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.976829 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.976736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-service-ca\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.977253 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.977226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-oauth-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.977381 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.977306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.977481 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.977459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-service-ca\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.977644 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.977623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a41dfb-e6fa-409c-ae78-feb7f5423642-trusted-ca-bundle\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.978844 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.978817 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-oauth-config\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.978949 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.978933 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92a41dfb-e6fa-409c-ae78-feb7f5423642-console-serving-cert\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:23.985351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:23.985333 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsjr\" (UniqueName: \"kubernetes.io/projected/92a41dfb-e6fa-409c-ae78-feb7f5423642-kube-api-access-nvsjr\") pod \"console-69dcc88757-8mlvr\" (UID: \"92a41dfb-e6fa-409c-ae78-feb7f5423642\") " pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:24.039263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:24.039226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:24.170841 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:24.170818 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dcc88757-8mlvr"] Jul 01 21:23:24.172965 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:24.172939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a41dfb_e6fa_409c_ae78_feb7f5423642.slice/crio-cce26e931caa25a1fb5dc27893023ac5964823fb0179b56edd179d80e0979980 WatchSource:0}: Error finding container cce26e931caa25a1fb5dc27893023ac5964823fb0179b56edd179d80e0979980: Status 404 returned error can't find the container with id cce26e931caa25a1fb5dc27893023ac5964823fb0179b56edd179d80e0979980 Jul 01 21:23:25.082052 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:25.082013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dcc88757-8mlvr" event={"ID":"92a41dfb-e6fa-409c-ae78-feb7f5423642","Type":"ContainerStarted","Data":"a33cb16cd681ce146bcf4b201b3615d0d9bc4f3df70912bedb8bc5e653490606"} Jul 01 21:23:25.082448 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:25.082078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dcc88757-8mlvr" event={"ID":"92a41dfb-e6fa-409c-ae78-feb7f5423642","Type":"ContainerStarted","Data":"cce26e931caa25a1fb5dc27893023ac5964823fb0179b56edd179d80e0979980"} Jul 01 21:23:25.099365 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:25.099319 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69dcc88757-8mlvr" podStartSLOduration=2.099301864 podStartE2EDuration="2.099301864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:25.098343685 +0000 UTC m=+704.152875767" watchObservedRunningTime="2026-07-01 21:23:25.099301864 +0000 UTC m=+704.153833947" Jul 01 21:23:28.052305 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:28.052275 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mqlw8" Jul 01 21:23:32.561859 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.561783 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4"] Jul 01 21:23:32.564789 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.564774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.566717 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.566695 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rsmk4\"" Jul 01 21:23:32.577128 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.577104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4"] Jul 01 21:23:32.648867 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.648830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.648867 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.648870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkftt\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-kube-api-access-rkftt\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.648929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.648983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.649028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.649079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649111 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.649107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649355 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.649132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04600e0c-3076-4eff-ac9d-aaa16329f37b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.649355 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.649223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749603 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749603 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkftt\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-kube-api-access-rkftt\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.749832 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.749829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04600e0c-3076-4eff-ac9d-aaa16329f37b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.750200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.750029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.750320 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.750297 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.750367 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.750337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.750612 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.750590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04600e0c-3076-4eff-ac9d-aaa16329f37b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.750729 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.750713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.752312 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.752284 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.752603 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.752584 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.757077 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.757050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.757531 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.757509 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkftt\" (UniqueName: \"kubernetes.io/projected/04600e0c-3076-4eff-ac9d-aaa16329f37b-kube-api-access-rkftt\") pod \"maas-default-gateway-openshift-default-56654dcf7-dh8x4\" (UID: \"04600e0c-3076-4eff-ac9d-aaa16329f37b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:32.874783 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:32.874690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:33.008044 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.008018 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4"] Jul 01 21:23:33.010020 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:33.009985 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04600e0c_3076_4eff_ac9d_aaa16329f37b.slice/crio-296e05114f598857d4747947ffdcc823fa3a78b6210a22a30872b6010f3e1abc WatchSource:0}: Error finding container 296e05114f598857d4747947ffdcc823fa3a78b6210a22a30872b6010f3e1abc: Status 404 returned error can't find the container with id 296e05114f598857d4747947ffdcc823fa3a78b6210a22a30872b6010f3e1abc Jul 01 21:23:33.012214 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.012158 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:23:33.012283 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.012266 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:23:33.012328 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.012312 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 01 21:23:33.123736 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.123687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" event={"ID":"04600e0c-3076-4eff-ac9d-aaa16329f37b","Type":"ContainerStarted","Data":"8c3d808139238699c12e7b69f15fc5ae0d2a3d6475910ce127fea04b35a16c46"} Jul 01 21:23:33.123922 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.123751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" event={"ID":"04600e0c-3076-4eff-ac9d-aaa16329f37b","Type":"ContainerStarted","Data":"296e05114f598857d4747947ffdcc823fa3a78b6210a22a30872b6010f3e1abc"} Jul 01 21:23:33.145442 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.145377 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" podStartSLOduration=1.145353722 podStartE2EDuration="1.145353722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:33.142950811 +0000 UTC m=+712.197482896" watchObservedRunningTime="2026-07-01 21:23:33.145353722 +0000 UTC m=+712.199885806" Jul 01 21:23:33.874961 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:33.874910 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:34.040354 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:34.040319 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:34.040550 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:34.040375 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:34.045003 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:34.044980 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:34.132608 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:34.132532 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69dcc88757-8mlvr" Jul 01 21:23:34.880399 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:34.880373 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:35.133936 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:35.133852 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:35.134916 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:35.134897 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-dh8x4" Jul 01 21:23:36.978028 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:36.977995 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:36.980434 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:36.980414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:36.982468 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:36.982451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jpc8h\"" Jul 01 21:23:36.982688 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:36.982668 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 01 21:23:36.992363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:36.992339 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:37.074988 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.074957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:37.089271 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.089237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.089271 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.089273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbfz\" (UniqueName: \"kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.190219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.190169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.190400 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.190226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbfz\" (UniqueName: \"kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.190877 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.190858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.197873 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.197845 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbfz\" (UniqueName: \"kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz\") pod \"limitador-limitador-64c94fd64f-k7x78\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.290627 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.290542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:37.461499 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:37.461473 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:37.463362 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:23:37.463319 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dcfbc3_78e9_4572_a7a2_0482889d4b9a.slice/crio-a9c66a1d1f61087a78dd162c68807ebd2c2417da83dfa4d0d03d4005d80e6b8c WatchSource:0}: Error finding container a9c66a1d1f61087a78dd162c68807ebd2c2417da83dfa4d0d03d4005d80e6b8c: Status 404 returned error can't find the container with id a9c66a1d1f61087a78dd162c68807ebd2c2417da83dfa4d0d03d4005d80e6b8c Jul 01 21:23:38.153757 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:38.153724 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" event={"ID":"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a","Type":"ContainerStarted","Data":"a9c66a1d1f61087a78dd162c68807ebd2c2417da83dfa4d0d03d4005d80e6b8c"} Jul 01 21:23:40.164841 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:40.164806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" event={"ID":"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a","Type":"ContainerStarted","Data":"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074"} Jul 01 21:23:40.165143 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:40.164893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:40.178871 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:40.178820 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" podStartSLOduration=1.575133515 podStartE2EDuration="4.178800098s" totalImagesPullingTime="2.603666583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:23:40.177957136 +0000 UTC m=+719.232489218" watchObservedRunningTime="2026-07-01 21:23:40.178800098 +0000 UTC m=+719.233332182" Jul 01 21:23:51.170363 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:51.170333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:54.362420 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.362390 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:54.362846 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.362603 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" podUID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" containerName="limitador" containerID="cri-o://e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074" gracePeriod=30 Jul 01 21:23:54.915847 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.915823 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:54.941940 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.941877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file\") pod \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " Jul 01 21:23:54.942048 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.942009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htbfz\" (UniqueName: \"kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz\") pod \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\" (UID: \"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a\") " Jul 01 21:23:54.942234 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.942214 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file" (OuterVolumeSpecName: "config-file") pod "a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" (UID: "a2dcfbc3-78e9-4572-a7a2-0482889d4b9a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 01 21:23:54.942294 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.942283 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-config-file\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:54.943925 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:54.943902 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz" (OuterVolumeSpecName: "kube-api-access-htbfz") pod "a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" (UID: "a2dcfbc3-78e9-4572-a7a2-0482889d4b9a"). InnerVolumeSpecName "kube-api-access-htbfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:23:55.043669 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.043638 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htbfz\" (UniqueName: \"kubernetes.io/projected/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a-kube-api-access-htbfz\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:23:55.230070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.229994 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" containerID="e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074" exitCode=0 Jul 01 21:23:55.230070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.230037 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" Jul 01 21:23:55.230070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.230043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" event={"ID":"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a","Type":"ContainerDied","Data":"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074"} Jul 01 21:23:55.230334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.230088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k7x78" event={"ID":"a2dcfbc3-78e9-4572-a7a2-0482889d4b9a","Type":"ContainerDied","Data":"a9c66a1d1f61087a78dd162c68807ebd2c2417da83dfa4d0d03d4005d80e6b8c"} Jul 01 21:23:55.230334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.230113 2577 scope.go:117] "RemoveContainer" containerID="e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074" Jul 01 21:23:55.239411 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.239392 2577 scope.go:117] "RemoveContainer" containerID="e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074" Jul 01 21:23:55.239706 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:23:55.239685 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074\": container with ID starting with e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074 not found: ID does not exist" containerID="e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074" Jul 01 21:23:55.239782 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.239719 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074"} err="failed to get container status \"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074\": rpc error: code = NotFound desc = could not find container \"e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074\": container with ID starting with e12092bd091e1663eb6b06f4cfba2d8fab682c4c24aab4eb20fe00cfc94c6074 not found: ID does not exist" Jul 01 21:23:55.250360 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.250337 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:55.254714 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.254696 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k7x78"] Jul 01 21:23:55.497056 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:23:55.496981 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" path="/var/lib/kubelet/pods/a2dcfbc3-78e9-4572-a7a2-0482889d4b9a/volumes" Jul 01 21:24:00.778525 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.778489 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-c7n56"] Jul 01 21:24:00.778984 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.778828 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" containerName="limitador" Jul 01 21:24:00.778984 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.778838 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" containerName="limitador" Jul 01 21:24:00.778984 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.778897 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dcfbc3-78e9-4572-a7a2-0482889d4b9a" containerName="limitador" Jul 01 21:24:00.784707 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.784684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:00.787367 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.787344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q9g9x\"" Jul 01 21:24:00.787525 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.787344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 01 21:24:00.788126 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.788103 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-c7n56"] Jul 01 21:24:00.898351 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.898315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxh6\" (UniqueName: \"kubernetes.io/projected/df717343-8927-4218-a2e6-33b86f344530-kube-api-access-shxh6\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:00.898541 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.898366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df717343-8927-4218-a2e6-33b86f344530-data\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:00.999901 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.999857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxh6\" (UniqueName: \"kubernetes.io/projected/df717343-8927-4218-a2e6-33b86f344530-kube-api-access-shxh6\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:00.999901 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:00.999909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df717343-8927-4218-a2e6-33b86f344530-data\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:01.000255 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:01.000240 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df717343-8927-4218-a2e6-33b86f344530-data\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:01.010829 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:01.010799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxh6\" (UniqueName: \"kubernetes.io/projected/df717343-8927-4218-a2e6-33b86f344530-kube-api-access-shxh6\") pod \"postgres-69fb4594cf-c7n56\" (UID: \"df717343-8927-4218-a2e6-33b86f344530\") " pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:01.096560 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:01.096482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:01.227045 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:01.227021 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-c7n56"] Jul 01 21:24:01.229119 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:24:01.229087 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf717343_8927_4218_a2e6_33b86f344530.slice/crio-a076c8abeadd97e0833e4ff704f3362c5acee9f07b7ea0503aed6e0d13780d9a WatchSource:0}: Error finding container a076c8abeadd97e0833e4ff704f3362c5acee9f07b7ea0503aed6e0d13780d9a: Status 404 returned error can't find the container with id a076c8abeadd97e0833e4ff704f3362c5acee9f07b7ea0503aed6e0d13780d9a Jul 01 21:24:01.258896 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:01.258864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-c7n56" event={"ID":"df717343-8927-4218-a2e6-33b86f344530","Type":"ContainerStarted","Data":"a076c8abeadd97e0833e4ff704f3362c5acee9f07b7ea0503aed6e0d13780d9a"} Jul 01 21:24:06.285271 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:06.285238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-c7n56" event={"ID":"df717343-8927-4218-a2e6-33b86f344530","Type":"ContainerStarted","Data":"08b90ea7dceb9d049b4976ef7fa34b8634b3487cf76ef2227d070b621a9d4df0"} Jul 01 21:24:06.285641 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:06.285382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:06.299279 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:06.299236 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-c7n56" podStartSLOduration=1.329850881 podStartE2EDuration="6.299217718s" totalImagesPullingTime="4.969366837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:24:06.298593489 +0000 UTC m=+745.353125572" watchObservedRunningTime="2026-07-01 21:24:06.299217718 +0000 UTC m=+745.353749802" Jul 01 21:24:12.319302 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:12.319277 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-c7n56" Jul 01 21:24:37.278607 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.278568 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 01 21:24:37.282994 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.282977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:37.285419 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.285397 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jul 01 21:24:37.285521 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.285407 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jul 01 21:24:37.285673 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.285659 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 01 21:24:37.285713 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.285671 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-lkrft\"" Jul 01 21:24:37.291257 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.291234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 01 21:24:37.320615 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.320588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpps7\" (UniqueName: \"kubernetes.io/projected/378450a3-21ab-4888-952c-d51ffa84aa60-kube-api-access-vpps7\") pod \"maas-keycloak-0\" (UID: \"378450a3-21ab-4888-952c-d51ffa84aa60\") " pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:37.421631 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.421594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpps7\" (UniqueName: \"kubernetes.io/projected/378450a3-21ab-4888-952c-d51ffa84aa60-kube-api-access-vpps7\") pod \"maas-keycloak-0\" (UID: \"378450a3-21ab-4888-952c-d51ffa84aa60\") " pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:37.430374 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.430344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpps7\" (UniqueName: \"kubernetes.io/projected/378450a3-21ab-4888-952c-d51ffa84aa60-kube-api-access-vpps7\") pod \"maas-keycloak-0\" (UID: \"378450a3-21ab-4888-952c-d51ffa84aa60\") " pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:37.593373 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.593284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:37.721706 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:37.721683 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 01 21:24:37.723843 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:24:37.723809 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378450a3_21ab_4888_952c_d51ffa84aa60.slice/crio-c03afb4cb31d6d3b88cd086086b7961739cb67227cd67ba2b09be208f97328ef WatchSource:0}: Error finding container c03afb4cb31d6d3b88cd086086b7961739cb67227cd67ba2b09be208f97328ef: Status 404 returned error can't find the container with id c03afb4cb31d6d3b88cd086086b7961739cb67227cd67ba2b09be208f97328ef Jul 01 21:24:38.435275 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:38.435238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"378450a3-21ab-4888-952c-d51ffa84aa60","Type":"ContainerStarted","Data":"c03afb4cb31d6d3b88cd086086b7961739cb67227cd67ba2b09be208f97328ef"} Jul 01 21:24:42.826425 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:42.826402 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 01 21:24:43.466576 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:43.466543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"378450a3-21ab-4888-952c-d51ffa84aa60","Type":"ContainerStarted","Data":"f4001ce51075cdf18c2d181b04156c2dbc52262547bc058cb8ef780834e830a0"} Jul 01 21:24:43.484503 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:43.484446 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=1.385632075 podStartE2EDuration="6.484426882s" totalImagesPullingTime="5.098794807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:24:43.483606152 +0000 UTC m=+782.538138235" watchObservedRunningTime="2026-07-01 21:24:43.484426882 +0000 UTC m=+782.538958963" Jul 01 21:24:43.594102 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:43.594062 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:43.595690 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:43.595656 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:44.594817 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:44.594765 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:45.594453 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:45.594408 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:46.594407 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:46.594363 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:47.594447 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:47.594396 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:47.594964 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:47.594887 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:48.593988 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:48.593940 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:49.594048 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:49.593992 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:50.594788 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:50.594738 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:51.594661 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:51.594616 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:52.594453 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:52.594407 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:53.594822 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:53.594666 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:54.594278 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:54.594225 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:55.594673 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:55.594632 2577 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="Get \"http://10.133.0.37:9000/health/started\": dial tcp 10.133.0.37:9000: connect: connection refused" Jul 01 21:24:56.718168 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:56.718131 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jul 01 21:24:56.734692 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:24:56.734635 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="378450a3-21ab-4888-952c-d51ffa84aa60" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 01 21:25:06.723742 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:06.723700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jul 01 21:25:13.992725 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:13.992686 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:14.000264 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.000239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.002456 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.002433 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 01 21:25:14.002695 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.002676 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 01 21:25:14.002789 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.002684 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pznks\"" Jul 01 21:25:14.009905 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.009880 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:14.084826 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.084791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztjj\" (UniqueName: \"kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.084993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.084906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.164065 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.164025 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64df85cb67-lj47p"] Jul 01 21:25:14.173890 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.173860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.175835 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.175808 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64df85cb67-lj47p"] Jul 01 21:25:14.185560 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.185538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.185708 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.185609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztjj\" (UniqueName: \"kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.185708 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.185648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jprt\" (UniqueName: \"kubernetes.io/projected/892c9b8e-884a-40fa-b53a-8de14944311d-kube-api-access-7jprt\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.185708 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.185694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/892c9b8e-884a-40fa-b53a-8de14944311d-webhook-cert\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.188275 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.188247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.193385 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.193355 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztjj\" (UniqueName: \"kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj\") pod \"maas-controller-64c4fc49f9-kbx2h\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.286890 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.286789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/892c9b8e-884a-40fa-b53a-8de14944311d-webhook-cert\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.286890 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.286885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jprt\" (UniqueName: \"kubernetes.io/projected/892c9b8e-884a-40fa-b53a-8de14944311d-kube-api-access-7jprt\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.289347 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.289322 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/892c9b8e-884a-40fa-b53a-8de14944311d-webhook-cert\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.294393 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.294369 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jprt\" (UniqueName: \"kubernetes.io/projected/892c9b8e-884a-40fa-b53a-8de14944311d-kube-api-access-7jprt\") pod \"maas-controller-64df85cb67-lj47p\" (UID: \"892c9b8e-884a-40fa-b53a-8de14944311d\") " pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.311928 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.311900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:14.472422 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.472386 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:14.475861 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:25:14.475832 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc462de1_8bae_44f4_a2bd_440dbd424ca0.slice/crio-a4e2cc443e6f2f7889ab5b94bad724d7ccc5395bb4c269077048f6b6d5530552 WatchSource:0}: Error finding container a4e2cc443e6f2f7889ab5b94bad724d7ccc5395bb4c269077048f6b6d5530552: Status 404 returned error can't find the container with id a4e2cc443e6f2f7889ab5b94bad724d7ccc5395bb4c269077048f6b6d5530552 Jul 01 21:25:14.485936 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.485916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:14.641384 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.641345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" event={"ID":"dc462de1-8bae-44f4-a2bd-440dbd424ca0","Type":"ContainerStarted","Data":"a4e2cc443e6f2f7889ab5b94bad724d7ccc5395bb4c269077048f6b6d5530552"} Jul 01 21:25:14.641876 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:14.641853 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64df85cb67-lj47p"] Jul 01 21:25:14.643377 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:25:14.643357 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892c9b8e_884a_40fa_b53a_8de14944311d.slice/crio-453ca52ce106827097b6d391c66f4e3cf5976fd2cb39075f923490ebee3bf8f3 WatchSource:0}: Error finding container 453ca52ce106827097b6d391c66f4e3cf5976fd2cb39075f923490ebee3bf8f3: Status 404 returned error can't find the container with id 453ca52ce106827097b6d391c66f4e3cf5976fd2cb39075f923490ebee3bf8f3 Jul 01 21:25:15.666219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:15.665497 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64df85cb67-lj47p" event={"ID":"892c9b8e-884a-40fa-b53a-8de14944311d","Type":"ContainerStarted","Data":"453ca52ce106827097b6d391c66f4e3cf5976fd2cb39075f923490ebee3bf8f3"} Jul 01 21:25:17.678857 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.678811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64df85cb67-lj47p" event={"ID":"892c9b8e-884a-40fa-b53a-8de14944311d","Type":"ContainerStarted","Data":"81ae2034de573c813d42fbeeb43fc4ab0539b22e834c8f2d22fa750e0abffea3"} Jul 01 21:25:17.679320 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.678904 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:17.680748 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.680709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" event={"ID":"dc462de1-8bae-44f4-a2bd-440dbd424ca0","Type":"ContainerStarted","Data":"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04"} Jul 01 21:25:17.680899 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.680798 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:17.711882 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.711825 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64df85cb67-lj47p" podStartSLOduration=0.849997734 podStartE2EDuration="3.711802354s" totalImagesPullingTime="2.86180462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:25:17.707222385 +0000 UTC m=+816.761754466" watchObservedRunningTime="2026-07-01 21:25:17.711802354 +0000 UTC m=+816.766334437" Jul 01 21:25:17.729507 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:17.729457 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" podStartSLOduration=1.709556906 podStartE2EDuration="4.729441256s" totalImagesPullingTime="3.01988435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:25:17.726414442 +0000 UTC m=+816.780946524" watchObservedRunningTime="2026-07-01 21:25:17.729441256 +0000 UTC m=+816.783973338" Jul 01 21:25:28.692287 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:28.692258 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:28.692738 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:28.692322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64df85cb67-lj47p" Jul 01 21:25:28.752098 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:28.752066 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:28.752336 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:28.752313 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" podUID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" containerName="manager" containerID="cri-o://03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04" gracePeriod=30 Jul 01 21:25:29.000136 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.000112 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:29.124990 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.124958 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wztjj\" (UniqueName: \"kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj\") pod \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " Jul 01 21:25:29.125194 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.125017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert\") pod \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\" (UID: \"dc462de1-8bae-44f4-a2bd-440dbd424ca0\") " Jul 01 21:25:29.127219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.127170 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc462de1-8bae-44f4-a2bd-440dbd424ca0" (UID: "dc462de1-8bae-44f4-a2bd-440dbd424ca0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 01 21:25:29.127320 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.127229 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj" (OuterVolumeSpecName: "kube-api-access-wztjj") pod "dc462de1-8bae-44f4-a2bd-440dbd424ca0" (UID: "dc462de1-8bae-44f4-a2bd-440dbd424ca0"). InnerVolumeSpecName "kube-api-access-wztjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 01 21:25:29.225566 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.225537 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc462de1-8bae-44f4-a2bd-440dbd424ca0-webhook-cert\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:25:29.225566 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.225563 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wztjj\" (UniqueName: \"kubernetes.io/projected/dc462de1-8bae-44f4-a2bd-440dbd424ca0-kube-api-access-wztjj\") on node \"ip-10-0-140-112.ec2.internal\" DevicePath \"\"" Jul 01 21:25:29.746201 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.746146 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" containerID="03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04" exitCode=0 Jul 01 21:25:29.746570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.746223 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" Jul 01 21:25:29.746570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.746231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" event={"ID":"dc462de1-8bae-44f4-a2bd-440dbd424ca0","Type":"ContainerDied","Data":"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04"} Jul 01 21:25:29.746570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.746287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64c4fc49f9-kbx2h" event={"ID":"dc462de1-8bae-44f4-a2bd-440dbd424ca0","Type":"ContainerDied","Data":"a4e2cc443e6f2f7889ab5b94bad724d7ccc5395bb4c269077048f6b6d5530552"} Jul 01 21:25:29.746570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.746309 2577 scope.go:117] "RemoveContainer" containerID="03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04" Jul 01 21:25:29.754791 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.754585 2577 scope.go:117] "RemoveContainer" containerID="03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04" Jul 01 21:25:29.754851 ip-10-0-140-112 kubenswrapper[2577]: E0701 21:25:29.754826 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04\": container with ID starting with 03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04 not found: ID does not exist" containerID="03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04" Jul 01 21:25:29.754893 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.754851 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04"} err="failed to get container status \"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04\": rpc error: code = NotFound desc = could not find container \"03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04\": container with ID starting with 03ca0b43b3240a61a829c5ef0c720d5fa018920ee17000da00680fe044c43d04 not found: ID does not exist" Jul 01 21:25:29.760619 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.760598 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:29.766410 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:29.766391 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64c4fc49f9-kbx2h"] Jul 01 21:25:31.496132 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:25:31.496097 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" path="/var/lib/kubelet/pods/dc462de1-8bae-44f4-a2bd-440dbd424ca0/volumes" Jul 01 21:35:39.781930 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:39.781859 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkckm_2a6a5dca-d629-4060-a7bf-7c44dbdb0bba/manager/0.log" Jul 01 21:35:39.890042 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:39.890017 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lnrxj_f3658e83-440b-48c1-ba75-3236bc6fd760/manager/0.log" Jul 01 21:35:39.999070 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:39.999035 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64df85cb67-lj47p_892c9b8e-884a-40fa-b53a-8de14944311d/manager/0.log" Jul 01 21:35:40.112263 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:40.112194 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qqx9d_1fae2f73-a3e4-4ff3-8419-fd931ce61125/server/0.log" Jul 01 21:35:40.254378 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:40.254350 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-78qzx_872757d0-baca-4bfa-a2c5-9e1ea4009289/manager/0.log" Jul 01 21:35:40.500866 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:40.500838 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d96b6c7-fz76w_de5798ac-b6be-4758-90a8-9c77abcd4dc3/copy-manifests/0.log" Jul 01 21:35:40.506233 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:40.506203 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d96b6c7-fz76w_de5798ac-b6be-4758-90a8-9c77abcd4dc3/manager/0.log" Jul 01 21:35:40.754233 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:40.754155 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-c7n56_df717343-8927-4218-a2e6-33b86f344530/postgres/0.log" Jul 01 21:35:42.115231 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:42.115200 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ll2w6_a909f286-59b4-47fa-9631-104db00cdd78/manager/0.log" Jul 01 21:35:42.446412 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:42.446385 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6d64x_366a100e-7550-48fc-b9b3-e6924c176960/registry-server/0.log" Jul 01 21:35:42.559033 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:42.559004 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mqlw8_3e0792f4-c321-423f-8eb2-4c7d68eb9f77/manager/0.log" Jul 01 21:35:42.782848 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:42.782766 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vwfvj_051b7308-0c2f-4085-9d55-b15e1ce84296/manager/0.log" Jul 01 21:35:43.110907 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:43.110827 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ndwdc_60c965a9-780d-4f35-8711-c7909a819ffa/istio-proxy/0.log" Jul 01 21:35:43.444632 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:43.444597 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb45b86b-lg2zb_a7771f04-6862-445d-96a3-6263fb781f5a/kube-auth-proxy/0.log" Jul 01 21:35:43.560077 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:43.560046 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-dh8x4_04600e0c-3076-4eff-ac9d-aaa16329f37b/istio-proxy/0.log" Jul 01 21:35:50.026909 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:50.026883 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gl4jg_55550433-0881-4a26-b1e8-13b7b9e6e937/global-pull-secret-syncer/0.log" Jul 01 21:35:50.192861 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:50.192830 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xxxwx_288a741e-d8d9-444e-91ce-2a4481b0ed30/konnectivity-agent/0.log" Jul 01 21:35:50.231388 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:50.231360 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-112.ec2.internal_ae7768ae7f0cea9300e6e78d611f1271/haproxy/0.log" Jul 01 21:35:54.539442 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:54.539414 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ll2w6_a909f286-59b4-47fa-9631-104db00cdd78/manager/0.log" Jul 01 21:35:54.624118 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:54.624085 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6d64x_366a100e-7550-48fc-b9b3-e6924c176960/registry-server/0.log" Jul 01 21:35:54.648702 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:54.648671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-mqlw8_3e0792f4-c321-423f-8eb2-4c7d68eb9f77/manager/0.log" Jul 01 21:35:54.694165 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:54.694128 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vwfvj_051b7308-0c2f-4085-9d55-b15e1ce84296/manager/0.log" Jul 01 21:35:56.700990 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.700914 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm4bn_8f08d2dc-39bb-42ba-bbeb-dff005d4be88/node-exporter/0.log" Jul 01 21:35:56.716976 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.716957 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm4bn_8f08d2dc-39bb-42ba-bbeb-dff005d4be88/kube-rbac-proxy/0.log" Jul 01 21:35:56.734291 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.734264 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm4bn_8f08d2dc-39bb-42ba-bbeb-dff005d4be88/init-textfile/0.log" Jul 01 21:35:56.827914 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.827887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-qgvzx_5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70/kube-rbac-proxy-main/0.log" Jul 01 21:35:56.847570 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.847545 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-qgvzx_5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70/kube-rbac-proxy-self/0.log" Jul 01 21:35:56.865361 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.865336 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-qgvzx_5a6bb4aa-8f9e-4c4b-b0d4-c6ca52791f70/openshift-state-metrics/0.log" Jul 01 21:35:56.914454 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.914420 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/prometheus/0.log" Jul 01 21:35:56.942106 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.942087 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/config-reloader/0.log" Jul 01 21:35:56.964367 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.964306 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/thanos-sidecar/0.log" Jul 01 21:35:56.982870 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:56.982849 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/kube-rbac-proxy-web/0.log" Jul 01 21:35:57.004294 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:57.004276 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/kube-rbac-proxy/0.log" Jul 01 21:35:57.023986 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:57.023965 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/kube-rbac-proxy-thanos/0.log" Jul 01 21:35:57.042709 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:57.042686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e79cb2e-0ecd-4dc1-9d12-9620ca46b630/init-config-reloader/0.log" Jul 01 21:35:57.124982 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:35:57.124936 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-wb2bp_cbca3372-01e5-4aa5-82cf-32fca4f20917/prometheus-operator-admission-webhook/0.log" Jul 01 21:36:00.147569 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.147536 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69dcc88757-8mlvr_92a41dfb-e6fa-409c-ae78-feb7f5423642/console/0.log" Jul 01 21:36:00.804212 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.804165 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq"] Jul 01 21:36:00.804596 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.804583 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" containerName="manager" Jul 01 21:36:00.804640 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.804598 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" containerName="manager" Jul 01 21:36:00.804674 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.804654 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc462de1-8bae-44f4-a2bd-440dbd424ca0" containerName="manager" Jul 01 21:36:00.807734 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.807719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:00.809928 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.809886 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-42f4z\"/\"kube-root-ca.crt\"" Jul 01 21:36:00.809928 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.809895 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-42f4z\"/\"openshift-service-ca.crt\"" Jul 01 21:36:00.810347 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.810329 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-42f4z\"/\"default-dockercfg-f6kw6\"" Jul 01 21:36:00.815796 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.815776 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq"] Jul 01 21:36:00.907220 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.907163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-lib-modules\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:00.907404 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.907255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-sys\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:00.907404 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.907317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-podres\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:00.907404 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.907377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw9xx\" (UniqueName: \"kubernetes.io/projected/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-kube-api-access-mw9xx\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:00.907521 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:00.907415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-proc\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.007862 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-podres\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw9xx\" (UniqueName: \"kubernetes.io/projected/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-kube-api-access-mw9xx\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-proc\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-lib-modules\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-sys\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.007996 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-podres\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008043 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.008031 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-proc\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008292 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.008038 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-sys\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.008292 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.008105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-lib-modules\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.015469 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.015441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw9xx\" (UniqueName: \"kubernetes.io/projected/1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70-kube-api-access-mw9xx\") pod \"perf-node-gather-daemonset-xl9rq\" (UID: \"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70\") " pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.118397 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.118305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.248639 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.248616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq"] Jul 01 21:36:01.250838 ip-10-0-140-112 kubenswrapper[2577]: W0701 21:36:01.250803 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e9ff3d9_0ab5_4876_8fcd_9b3004ae2b70.slice/crio-60db25fd88b91d4b7c1b1436e94252ac8b6847b0ebd69ebc63fb9b38c077aa69 WatchSource:0}: Error finding container 60db25fd88b91d4b7c1b1436e94252ac8b6847b0ebd69ebc63fb9b38c077aa69: Status 404 returned error can't find the container with id 60db25fd88b91d4b7c1b1436e94252ac8b6847b0ebd69ebc63fb9b38c077aa69 Jul 01 21:36:01.650073 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.650032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" event={"ID":"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70","Type":"ContainerStarted","Data":"a1021eaae69396d4c88221178bd6efccc4c920521b3f2bb78b5c697d66d0a230"} Jul 01 21:36:01.650303 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.650087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" event={"ID":"1e9ff3d9-0ab5-4876-8fcd-9b3004ae2b70","Type":"ContainerStarted","Data":"60db25fd88b91d4b7c1b1436e94252ac8b6847b0ebd69ebc63fb9b38c077aa69"} Jul 01 21:36:01.650303 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.650168 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:01.664553 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.664492 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" podStartSLOduration=1.664477056 podStartE2EDuration="1.664477056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:36:01.663210443 +0000 UTC m=+1460.717742522" watchObservedRunningTime="2026-07-01 21:36:01.664477056 +0000 UTC m=+1460.719009177" Jul 01 21:36:01.937200 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.937105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wncw6_b5557a0e-5758-4f66-951b-8b7d55673f07/dns/0.log" Jul 01 21:36:01.954888 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:01.954867 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wncw6_b5557a0e-5758-4f66-951b-8b7d55673f07/kube-rbac-proxy/0.log" Jul 01 21:36:02.011359 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:02.011331 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zc8cn_b8d70855-fe25-4e82-9b5f-694d5890e5be/dns-node-resolver/0.log" Jul 01 21:36:02.543607 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:02.543573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6vwh_c9b97c52-11f1-4a3e-9d3c-f75126b58de3/node-ca/0.log" Jul 01 21:36:03.456798 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:03.456763 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ndwdc_60c965a9-780d-4f35-8711-c7909a819ffa/istio-proxy/0.log" Jul 01 21:36:03.528403 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:03.528374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb45b86b-lg2zb_a7771f04-6862-445d-96a3-6263fb781f5a/kube-auth-proxy/0.log" Jul 01 21:36:03.552630 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:03.552604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-dh8x4_04600e0c-3076-4eff-ac9d-aaa16329f37b/istio-proxy/0.log" Jul 01 21:36:04.138047 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:04.138017 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cvzw6_a87629f6-6a84-4c9e-9a9d-730de62f9df1/serve-healthcheck-canary/0.log" Jul 01 21:36:04.621983 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:04.621952 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9qthx_ec91955f-f427-4415-9149-53eb1e51f071/kube-rbac-proxy/0.log" Jul 01 21:36:04.638742 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:04.638719 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9qthx_ec91955f-f427-4415-9149-53eb1e51f071/exporter/0.log" Jul 01 21:36:04.654659 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:04.654627 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9qthx_ec91955f-f427-4415-9149-53eb1e51f071/extractor/0.log" Jul 01 21:36:06.840369 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:06.840340 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkckm_2a6a5dca-d629-4060-a7bf-7c44dbdb0bba/manager/0.log" Jul 01 21:36:06.865993 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:06.865967 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lnrxj_f3658e83-440b-48c1-ba75-3236bc6fd760/manager/0.log" Jul 01 21:36:06.888137 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:06.888105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64df85cb67-lj47p_892c9b8e-884a-40fa-b53a-8de14944311d/manager/0.log" Jul 01 21:36:06.909563 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:06.909533 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qqx9d_1fae2f73-a3e4-4ff3-8419-fd931ce61125/server/0.log" Jul 01 21:36:06.954851 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:06.954803 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-78qzx_872757d0-baca-4bfa-a2c5-9e1ea4009289/manager/0.log" Jul 01 21:36:07.018391 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:07.018359 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d96b6c7-fz76w_de5798ac-b6be-4758-90a8-9c77abcd4dc3/manager/0.log" Jul 01 21:36:07.038073 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:07.038045 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d96b6c7-fz76w_de5798ac-b6be-4758-90a8-9c77abcd4dc3/copy-manifests/0.log" Jul 01 21:36:07.123494 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:07.123417 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-c7n56_df717343-8927-4218-a2e6-33b86f344530/postgres/0.log" Jul 01 21:36:07.664623 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:07.664597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-42f4z/perf-node-gather-daemonset-xl9rq" Jul 01 21:36:10.338507 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:10.338482 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ndwdc_60c965a9-780d-4f35-8711-c7909a819ffa/istio-proxy/0.log" Jul 01 21:36:10.401171 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:10.401142 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb45b86b-lg2zb_a7771f04-6862-445d-96a3-6263fb781f5a/kube-auth-proxy/0.log" Jul 01 21:36:10.419219 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:10.419195 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-dh8x4_04600e0c-3076-4eff-ac9d-aaa16329f37b/istio-proxy/0.log" Jul 01 21:36:11.727574 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:11.727540 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ndwdc_60c965a9-780d-4f35-8711-c7909a819ffa/istio-proxy/0.log" Jul 01 21:36:11.783715 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:11.783689 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb45b86b-lg2zb_a7771f04-6862-445d-96a3-6263fb781f5a/kube-auth-proxy/0.log" Jul 01 21:36:11.810417 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:11.810374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-dh8x4_04600e0c-3076-4eff-ac9d-aaa16329f37b/istio-proxy/0.log" Jul 01 21:36:15.346478 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.346446 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/kube-multus-additional-cni-plugins/0.log" Jul 01 21:36:15.363310 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.363281 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/egress-router-binary-copy/0.log" Jul 01 21:36:15.379782 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.379760 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/cni-plugins/0.log" Jul 01 21:36:15.397796 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.397773 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/bond-cni-plugin/0.log" Jul 01 21:36:15.416465 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.416447 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/routeoverride-cni/0.log" Jul 01 21:36:15.433486 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.433461 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/whereabouts-cni-bincopy/0.log" Jul 01 21:36:15.450975 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.450954 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9fxl_ab94aa7d-dcdf-4038-a02d-eb23db442513/whereabouts-cni/0.log" Jul 01 21:36:15.521061 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.521027 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r96q2_bd66d419-d121-470c-a388-4d3b1ff97ea4/kube-multus/0.log" Jul 01 21:36:15.613734 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.613664 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q79ff_a911bcd8-9282-410b-8039-b9d043324899/network-metrics-daemon/0.log" Jul 01 21:36:15.629292 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:15.629268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q79ff_a911bcd8-9282-410b-8039-b9d043324899/kube-rbac-proxy/0.log" Jul 01 21:36:16.578823 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.578794 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/ovn-controller/0.log" Jul 01 21:36:16.594362 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.594337 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/ovn-acl-logging/0.log" Jul 01 21:36:16.615512 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.615486 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/kube-rbac-proxy-node/0.log" Jul 01 21:36:16.634516 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.634485 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/kube-rbac-proxy-ovn-metrics/0.log" Jul 01 21:36:16.647800 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.647774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/northd/0.log" Jul 01 21:36:16.664799 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.664774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/nbdb/0.log" Jul 01 21:36:16.680723 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.680704 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/sbdb/0.log" Jul 01 21:36:16.717334 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:16.717305 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dm7qs_813b4b2e-b459-4809-b455-f29d00c476e0/ovnkube-controller/0.log" Jul 01 21:36:18.022344 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:18.022312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qnl77_c40564fb-5f5a-4698-83fe-e5d0bc2c1d58/network-check-target-container/0.log" Jul 01 21:36:19.011066 ip-10-0-140-112 kubenswrapper[2577]: I0701 21:36:19.011039 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-28nm5_00ca11bc-d58b-4fbb-871b-f5de4c360041/iptables-alerter/0.log"