Jul 02 20:33:02.004816 ip-10-0-140-111 systemd[1]: Starting Kubernetes Kubelet... Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.406026 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412108 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412131 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412135 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412138 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412141 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412151 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412154 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.434355 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412159 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412162 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412165 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412167 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412171 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412174 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412177 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412180 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412183 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412187 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412189 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412192 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412195 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412197 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412200 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412204 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412207 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412210 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412213 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412215 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.435605 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412218 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412221 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412224 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412227 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412229 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412232 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412235 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412237 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412240 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412243 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412245 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412251 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412255 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412258 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412261 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412264 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412268 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412270 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412273 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.436809 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412275 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412278 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412281 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412284 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412287 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412290 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412293 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412295 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412298 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412301 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412304 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412307 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412311 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412315 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412319 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412322 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412327 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412330 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412333 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.437720 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412336 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412339 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412342 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412345 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412348 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412351 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412354 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412357 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412360 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412363 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412365 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412368 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412371 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412373 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412376 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412379 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412381 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412384 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412386 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412389 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.438644 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412391 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412393 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412397 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412400 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412403 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412406 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412409 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412411 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412414 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412416 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412419 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412422 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412426 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.412430 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413312 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413329 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413333 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413336 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413339 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.439783 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413343 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413346 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413349 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413352 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413355 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413358 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413361 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413365 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413368 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413370 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413373 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413376 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413379 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413382 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413385 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413388 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413390 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413393 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413396 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.440682 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413399 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413401 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413404 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413406 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413410 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413413 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413415 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413418 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413420 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413424 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413427 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413429 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413434 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413439 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413442 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413445 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413463 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413466 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413469 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413472 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.441584 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413475 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413477 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413480 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413484 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413487 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413489 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413492 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413495 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413497 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413500 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413503 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413506 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413509 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413511 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413514 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413516 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413520 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413523 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413525 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.442704 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413528 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413531 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413534 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413538 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413540 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413543 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413545 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413548 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413550 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413554 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413559 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413563 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413566 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413570 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413573 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413577 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413579 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413582 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413585 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413587 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.443592 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413590 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413593 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413596 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413599 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413602 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413605 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413608 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413612 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413615 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413618 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413621 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413625 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413627 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413630 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413633 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.413636 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414070 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414078 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414083 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.444487 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414087 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414094 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414097 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414100 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414104 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414106 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414109 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414112 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414114 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414117 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414120 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414122 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414125 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414128 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414131 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414134 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414137 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414140 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414142 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414144 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.445366 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414147 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414150 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414152 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414155 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414158 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414160 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414163 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414165 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414170 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414176 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414179 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414181 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414185 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414191 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414194 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414197 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414200 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414203 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414205 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.446250 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414208 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414211 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414214 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414216 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414219 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414222 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414225 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414227 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414230 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414233 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414235 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414238 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414241 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414243 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414246 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414249 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414252 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414255 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414258 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414260 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414263 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.447129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414265 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414268 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414271 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414274 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414276 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414281 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414284 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414287 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414289 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414292 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414294 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414297 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414300 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414302 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414305 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414309 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414312 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414315 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414318 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.448026 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414320 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414323 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414325 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414328 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414330 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414333 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414335 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414338 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414341 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414343 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414346 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414348 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414351 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414353 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414356 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414359 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.414362 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414444 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414471 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414481 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414486 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 20:33:02.448903 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414490 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414494 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414499 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414503 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414507 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414510 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414513 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414521 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414525 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414528 2569 flags.go:64] FLAG: --cgroup-root="" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414531 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414534 2569 flags.go:64] FLAG: --client-ca-file="" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414537 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414540 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414544 2569 flags.go:64] FLAG: --cluster-domain="" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414547 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414550 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414553 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414557 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414561 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414564 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414568 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414594 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414599 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 02 20:33:02.449788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414602 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414606 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414609 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414612 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414618 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414622 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414625 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414628 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414631 2569 flags.go:64] FLAG: --enable-server="true" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414634 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414639 2569 flags.go:64] FLAG: --event-burst="100" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414643 2569 flags.go:64] FLAG: --event-qps="50" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414646 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414648 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414652 2569 flags.go:64] FLAG: --eviction-hard="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414657 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414660 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414664 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414667 2569 flags.go:64] FLAG: --eviction-soft="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414670 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414673 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414676 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414679 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414682 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414685 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 20:33:02.450878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414688 2569 flags.go:64] FLAG: --feature-gates="" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414692 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414695 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414699 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414702 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414705 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414708 2569 flags.go:64] FLAG: --help="false" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414711 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414714 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414717 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414720 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414723 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414729 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414732 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414735 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414738 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414741 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414744 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414747 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414750 2569 flags.go:64] FLAG: --kube-reserved="" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414753 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414756 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414760 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414764 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 20:33:02.452011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414767 2569 flags.go:64] FLAG: --lock-file="" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414769 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414772 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414775 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414782 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414785 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414788 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414791 2569 flags.go:64] FLAG: --logging-format="text" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414793 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414797 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414800 2569 flags.go:64] FLAG: --manifest-url="" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414803 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414812 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414815 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414819 2569 flags.go:64] FLAG: --max-pods="110" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414822 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414825 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414828 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414831 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414835 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414839 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414842 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414852 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414856 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414859 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 20:33:02.455715 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414862 2569 flags.go:64] FLAG: --pod-cidr="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414865 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414871 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414874 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414877 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414883 2569 flags.go:64] FLAG: --port="10250" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414886 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414889 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0660e5005b67412cf" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414893 2569 flags.go:64] FLAG: --qos-reserved="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414896 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414900 2569 flags.go:64] FLAG: --register-node="true" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414903 2569 flags.go:64] FLAG: --register-with-taints="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414907 2569 flags.go:64] FLAG: --registry-burst="10" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414909 2569 flags.go:64] FLAG: --registry-qps="5" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414912 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414915 2569 flags.go:64] FLAG: --reserved-memory="" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414924 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414927 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414931 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414934 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414937 2569 flags.go:64] FLAG: --runonce="false" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414940 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414943 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414946 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414949 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 20:33:02.456832 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414952 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414955 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414958 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414963 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414966 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414969 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414972 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414975 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414978 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414981 2569 flags.go:64] FLAG: --system-cgroups="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414984 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414990 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414994 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.414997 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415001 2569 flags.go:64] FLAG: --tls-min-version="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415003 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415006 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415009 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415012 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415015 2569 flags.go:64] FLAG: --v="2" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415020 2569 flags.go:64] FLAG: --version="false" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415024 2569 flags.go:64] FLAG: --vmodule="" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415028 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.415031 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415165 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.536147 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415169 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.487156 ip-10-0-140-111 systemd[1]: Started Kubernetes Kubelet. Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415172 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415176 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415180 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415183 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415186 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415188 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415191 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415194 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415197 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415229 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415234 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415239 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415242 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415245 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415248 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415251 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415254 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415256 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415260 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.537703 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415263 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415266 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415269 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415272 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415275 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415279 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415282 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415286 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415289 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415291 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415294 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415297 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415299 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415302 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415305 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415308 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415310 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415313 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415316 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415319 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.538893 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415322 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415324 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415328 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415331 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415333 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415337 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415339 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415342 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415345 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415347 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415350 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415354 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415356 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415359 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415361 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415364 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415366 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415369 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415371 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415374 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.539977 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415376 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415379 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415381 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415384 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415387 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415390 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415392 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415395 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415397 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415400 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415402 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415406 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415408 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415411 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415415 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415418 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415420 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415423 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415428 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.540927 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415432 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415435 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415437 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415440 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415445 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415463 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415467 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415469 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415472 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415475 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415477 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415480 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415483 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415485 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415488 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415490 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415493 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415496 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.542061 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.415499 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.416234 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.424343 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.424370 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.425183 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.425198 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425272 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425280 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425284 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425288 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425291 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425295 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425298 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425302 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425305 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.542973 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425308 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425311 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425314 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425317 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425320 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425324 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425327 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425329 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425332 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425334 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425337 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425340 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425342 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425345 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425348 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425350 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425353 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425356 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425358 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425361 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.543850 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425364 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425366 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425369 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425372 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425374 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425377 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425379 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425382 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425384 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425387 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425395 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425399 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425401 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425404 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425407 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425409 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425414 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425418 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425421 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.544975 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425424 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425427 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425430 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425433 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425436 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425439 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425441 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425445 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425468 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425474 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425478 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425482 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425492 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425496 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425501 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425506 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425509 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425511 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425514 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.546075 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425517 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425519 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425522 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425525 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425532 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425535 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425537 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425540 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425543 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425545 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425548 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425552 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425555 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425559 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425562 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425565 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425568 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425571 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425573 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.546946 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425576 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425579 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425582 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425585 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425587 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425590 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425592 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425595 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425597 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425601 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425604 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425606 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425609 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.425615 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425746 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:33:02.547987 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425751 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425754 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425758 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425763 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425766 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425769 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425771 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425774 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425777 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425779 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425784 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425787 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425790 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425793 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425796 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425798 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425802 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425805 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425808 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:33:02.548661 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425811 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425814 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425817 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425820 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425823 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425825 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425828 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425831 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425834 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425836 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425839 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425842 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425845 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425847 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425850 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425854 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425858 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425860 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425863 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425866 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:33:02.549734 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425868 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425871 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425874 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425876 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425879 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425881 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425884 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425887 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425889 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425892 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425894 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425897 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425900 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425902 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425905 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425907 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425910 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425912 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425915 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425917 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:33:02.550641 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425920 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425923 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425925 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425928 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425930 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425933 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425936 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425940 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425944 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425947 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425949 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425952 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425955 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425957 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425960 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425962 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425965 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425967 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425970 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425972 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:33:02.551735 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425975 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425979 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425981 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425984 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425986 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425989 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425991 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425994 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425996 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.425999 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426002 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426004 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426007 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426009 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426012 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426015 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426017 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426020 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:33:02.552642 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:02.426022 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.426030 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.426170 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.432207 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.433772 2569 server.go:1019] "Starting client certificate rotation" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.433910 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.433963 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.436296 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.437483 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.447861 2569 log.go:25] "Validated CRI v1 runtime API" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.452906 2569 log.go:25] "Validated CRI v1 image API" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.454294 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 20:33:02.553733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.456781 2569 fs.go:135] Filesystem UUIDs: map[0372cf81-040e-4a68-b7f5-0d999ee50f4b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bfe8c835-bb44-4fd3-a936-bb83825e41b6:/dev/nvme0n1p3] Jul 02 20:33:02.554395 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.456802 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.463192 2569 manager.go:216] Machine: {Timestamp:2026-07-02 20:33:02.461748819 +0000 UTC m=+0.330803545 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099660 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22ddc999fd6ec861b1a066b8089416 SystemUUID:ec22ddc9-99fd-6ec8-61b1-a066b8089416 BootID:6bef5363-d1c8-4f16-a7c1-79e0e849ce43 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:e8:0d:7b:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:e8:0d:7b:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:26:ae:7a:45:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.463330 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.463502 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.466577 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.466618 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.467618 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.467631 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 20:33:02.554614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.467641 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.468343 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.469909 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.473524 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.473678 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474422 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474445 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474479 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474493 2569 kubelet.go:398] "Adding apiserver pod source" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474506 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474946 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.474965 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.477939 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.480323 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.480635 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.480671 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481854 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481869 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481877 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481883 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481891 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481897 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481906 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481911 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481919 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481924 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481934 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.481941 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.482755 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.482766 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.486286 2569 server.go:1268] "Started kubelet" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.486288 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.486428 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.486594 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.486679 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.486729 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.486757 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.488922 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 20:33:02.558797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.489527 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.492911 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-111.ec2.internal.18be933a9ff42b73 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-111.ec2.internal,UID:ip-10-0-140-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-111.ec2.internal,},FirstTimestamp:2026-07-02 20:33:02.486260595 +0000 UTC m=+0.355315320,LastTimestamp:2026-07-02 20:33:02.486260595 +0000 UTC m=+0.355315320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-111.ec2.internal,}" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.494611 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.496693 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.497095 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.498938 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499679 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499688 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499709 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499810 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499818 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499846 2569 factory.go:55] Registering systemd factory Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.499859 2569 factory.go:223] Registration of the systemd container factory successfully Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500126 2569 factory.go:153] Registering CRI-O factory Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500143 2569 factory.go:223] Registration of the crio container factory successfully Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500203 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500234 2569 factory.go:103] Registering Raw factory Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500249 2569 manager.go:1196] Started watching for new ooms in manager Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.500676 2569 manager.go:349] Starting recovery of all containers Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.501808 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.502093 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.512398 2569 manager.go:354] Recovery completed Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.516537 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4jchm" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.522140 2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.527169 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.528026 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4jchm" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.531176 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.531216 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.531227 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.532085 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.532097 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.532119 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:33:02.560364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.533350 2569 policy_none.go:49] "None policy: Start" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.533364 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.533377 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.534490 2569 policy_none.go:47] "Start" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.535404 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.536530 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.536554 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.536563 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.599545 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617266 2569 manager.go:337] "Starting Device Plugin manager" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.617304 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617315 2569 server.go:87] "Starting device plugin registration server" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617673 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617688 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617775 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617903 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.617912 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.618610 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 20:33:02.641955 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.618658 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.710903 2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.718029 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.721830 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.721863 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.721876 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.721910 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.729276 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.729305 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-111.ec2.internal\": node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.753542 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.749912 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.777587 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.774020 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 20:33:02.777587 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.775368 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 20:33:02.777587 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.775410 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 20:33:02.777587 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.775443 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 20:33:02.777587 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.775502 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 20:33:02.778027 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.777889 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.850785 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.875947 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"] Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.876032 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.877027 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.877059 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.877075 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.878666 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.878768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.878792 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879387 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879395 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879411 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879417 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879428 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.879420 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.880857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.880897 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.881600 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.881620 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.881634 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.907208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.907245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:02.907269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78e29df1a4e6bf20e982f0f14de11bd1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"78e29df1a4e6bf20e982f0f14de11bd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.909536 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal" Jul 02 20:33:02.919525 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.914084 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:02.951631 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78e29df1a4e6bf20e982f0f14de11bd1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"78e29df1a4e6bf20e982f0f14de11bd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78e29df1a4e6bf20e982f0f14de11bd1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"78e29df1a4e6bf20e982f0f14de11bd1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.039479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.008240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6394b7562ea99074d8a3f3fd7a1feb82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"6394b7562ea99074d8a3f3fd7a1feb82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.149888 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.052191 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.153639 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.152640 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.274996 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.235270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.274996 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.237397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.274996 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.252822 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.403161 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.353649 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.403161 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.353860 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.432710 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.432859 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.432887 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.453949 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.531238 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 20:28:02 +0000 UTC" deadline="2028-01-15 16:18:35.867554994 +0000 UTC" Jul 02 20:33:03.538932 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.531285 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13483h45m32.336276129s" Jul 02 20:33:03.629307 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:03.554390 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Jul 02 20:33:03.629307 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.628495 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:03.707431 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.699535 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.710914 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.709734 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:33:03.710914 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.710650 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Jul 02 20:33:03.789687 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.720335 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:33:03.940608 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:03.808597 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e29df1a4e6bf20e982f0f14de11bd1.slice/crio-f94c37d6d4d63173919e9076e63532f5218f3691ed1eabe4cf0a042222b4ebb4 WatchSource:0}: Error finding container f94c37d6d4d63173919e9076e63532f5218f3691ed1eabe4cf0a042222b4ebb4: Status 404 returned error can't find the container with id f94c37d6d4d63173919e9076e63532f5218f3691ed1eabe4cf0a042222b4ebb4 Jul 02 20:33:03.940608 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:03.809124 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6394b7562ea99074d8a3f3fd7a1feb82.slice/crio-5057165158f2b31d7fff605d7b4f99bd0cce0cb399e4b29350d4cdcdb64a47f0 WatchSource:0}: Error finding container 5057165158f2b31d7fff605d7b4f99bd0cce0cb399e4b29350d4cdcdb64a47f0: Status 404 returned error can't find the container with id 5057165158f2b31d7fff605d7b4f99bd0cce0cb399e4b29350d4cdcdb64a47f0 Jul 02 20:33:03.940608 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:03.813696 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:33:04.101524 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.101490 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:04.476219 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.476130 2569 apiserver.go:52] "Watching apiserver" Jul 02 20:33:04.488894 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.488831 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 20:33:04.489965 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.489929 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-7f85s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","openshift-multus/multus-additional-cni-plugins-n9d5h","openshift-multus/multus-r7jnz","openshift-multus/network-metrics-daemon-hn2fv","openshift-network-diagnostics/network-check-target-w5ssn","openshift-ovn-kubernetes/ovnkube-node-7pb47","kube-system/konnectivity-agent-jcjrj","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf","openshift-cluster-node-tuning-operator/tuned-nwjhk","openshift-dns/node-resolver-tnqkm","openshift-network-operator/iptables-alerter-ctvj2","kube-system/global-pull-secret-syncer-qdqwm"] Jul 02 20:33:04.492357 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.492327 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.495708 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.495662 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.496912 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.496695 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.496912 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.496795 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.497190 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.496696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gg7c4\"" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.497471 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.497533 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.497474 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.497816 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.498022 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.498879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.498269 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.499488 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.498955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.499488 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.499148 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gx6tw\"" Jul 02 20:33:04.499488 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.499319 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 20:33:04.500109 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.500070 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.500409 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.500190 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 20:33:04.500678 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.500642 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 20:33:04.501086 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.500841 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 20:33:04.501086 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.500910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 20:33:04.501747 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.501581 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x7j6k\"" Jul 02 20:33:04.501747 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.501612 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.501972 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.501955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.502533 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.502507 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 20:33:04.502777 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.502750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.502983 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.502883 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:04.502983 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.502922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fdqfk\"" Jul 02 20:33:04.504732 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.504712 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:04.504829 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.504798 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:04.506452 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.506329 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.506452 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.506404 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:04.507916 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.507898 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.510050 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.510026 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.510748 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.510725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m2hn4\"" Jul 02 20:33:04.511133 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.511115 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.511312 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.511289 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.511513 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.511496 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 20:33:04.511944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.511921 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:33:04.514945 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.513601 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b6w8h\"" Jul 02 20:33:04.514945 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.513998 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 20:33:04.514945 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.514202 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 20:33:04.516773 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.516037 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517054 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-ovn\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517104 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bde5f90f-0617-41bf-b36e-8ecaadc1db67-iptables-alerter-script\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bde5f90f-0617-41bf-b36e-8ecaadc1db67-host-slash\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-socket-dir-parent\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-multus\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-kubelet\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-netns\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517371 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-netd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517404 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-os-release\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517531 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-system-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-slash\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnb56\" (UniqueName: \"kubernetes.io/projected/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-kube-api-access-tnb56\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517758 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-os-release\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cni-binary-copy\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-hostroot\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517830 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-daemon-config\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-var-lib-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-bin\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.517944 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-env-overrides\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.517959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cnibin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-systemd-units\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518040 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-script-lib\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-bin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-log-socket\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bbc40-7268-4a24-8a03-401696bbf923-ovn-node-metrics-cert\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pt82\" (UniqueName: \"kubernetes.io/projected/059bbc40-7268-4a24-8a03-401696bbf923-kube-api-access-6pt82\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-system-cni-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-netns\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-conf-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518386 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-multus-certs\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-systemd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.518610 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-node-log\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-k8s-cni-cncf-io\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-etc-kubernetes\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnx86\" (UniqueName: \"kubernetes.io/projected/35cc59a0-b27b-4c94-bed7-fe4459967ed4-kube-api-access-mnx86\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-kubelet\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-etc-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518645 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-config\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9wwg\" (UniqueName: \"kubernetes.io/projected/bde5f90f-0617-41bf-b36e-8ecaadc1db67-kube-api-access-h9wwg\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cnibin\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.518747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.519302 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.519287 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.519919 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.519307 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.519919 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.519316 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-md9hq\"" Jul 02 20:33:04.520803 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.520659 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.520803 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.520713 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbtsz\"" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.521535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.521835 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.523760 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.524770 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.526025 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 20:33:04.526499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.526031 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2z2jx\"" Jul 02 20:33:04.534076 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.534021 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz8kx" Jul 02 20:33:04.542171 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.542139 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz8kx" Jul 02 20:33:04.601196 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.601161 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 20:33:04.619842 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-k8s-cni-cncf-io\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-systemd\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-multus-certs\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-etc-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9wwg\" (UniqueName: \"kubernetes.io/projected/bde5f90f-0617-41bf-b36e-8ecaadc1db67-kube-api-access-h9wwg\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cnibin\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619967 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-k8s-cni-cncf-io\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.619987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-etc-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.620052 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620027 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-daemon-config\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bde5f90f-0617-41bf-b36e-8ecaadc1db67-iptables-alerter-script\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-multus-certs\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bde5f90f-0617-41bf-b36e-8ecaadc1db67-host-slash\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-multus\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-kubelet\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-sys-fs\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysconfig\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-registration-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-netns\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-host\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqrb\" (UniqueName: \"kubernetes.io/projected/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-kube-api-access-4jqrb\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620339 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dd58267-4611-45e1-87c2-673d21df0762-konnectivity-ca\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.620491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-slash\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cni-binary-copy\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-var-lib-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cnibin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-sys\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-systemd-units\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-bin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-socket-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-kubernetes\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620715 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-daemon-config\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bde5f90f-0617-41bf-b36e-8ecaadc1db67-host-slash\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f096e16-c852-477d-bcac-de5ae629e0be-hosts-file\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620777 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cnibin\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.621178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-log-socket\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bbc40-7268-4a24-8a03-401696bbf923-ovn-node-metrics-cert\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620837 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-multus\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pt82\" (UniqueName: \"kubernetes.io/projected/059bbc40-7268-4a24-8a03-401696bbf923-kube-api-access-6pt82\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-kubelet\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620880 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-netns\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-run-netns\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cnibin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620955 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-systemd-units\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-kubelet-config\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620708 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bde5f90f-0617-41bf-b36e-8ecaadc1db67-iptables-alerter-script\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.621870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621011 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-netns\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.620707 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621022 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-log-socket\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-etc-kubernetes\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621057 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-slash\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnx86\" (UniqueName: \"kubernetes.io/projected/35cc59a0-b27b-4c94-bed7-fe4459967ed4-kube-api-access-mnx86\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621112 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-etc-kubernetes\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621132 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-var-lib-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621130 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-host-var-lib-cni-bin\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-conf\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621182 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cc59a0-b27b-4c94-bed7-fe4459967ed4-cni-binary-copy\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-kubelet\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-config\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-kubelet\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.622642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621399 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rd7\" (UniqueName: \"kubernetes.io/projected/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-kube-api-access-q8rd7\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-hostroot\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-var-lib-kubelet\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gr2m\" (UniqueName: \"kubernetes.io/projected/7f096e16-c852-477d-bcac-de5ae629e0be-kube-api-access-8gr2m\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621596 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-hostroot\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621627 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621649 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-ovn\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621680 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-ovn\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-config\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621818 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-socket-dir-parent\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-run\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.623191 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.621982 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-socket-dir-parent\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-dbus\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-netd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-netd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622159 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622170 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-os-release\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-os-release\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-system-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f096e16-c852-477d-bcac-de5ae629e0be-tmp-dir\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb56\" (UniqueName: \"kubernetes.io/projected/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-kube-api-access-tnb56\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-os-release\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.623810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-system-cni-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mjj\" (UniqueName: \"kubernetes.io/projected/c475ceb4-f624-47b3-9f44-37a7d7aab228-kube-api-access-r7mjj\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622347 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-modprobe-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622424 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-os-release\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-host\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-serviceca\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622555 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-bin\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-env-overrides\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-lib-modules\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-tmp\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6j9m\" (UniqueName: \"kubernetes.io/projected/271dce04-514f-4da5-9e92-c4acc6811b50-kube-api-access-b6j9m\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.624451 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-script-lib\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-etc-selinux\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622745 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dd58267-4611-45e1-87c2-673d21df0762-agent-certs\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-system-cni-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622779 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-cni-bin\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-conf-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-device-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622854 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cc59a0-b27b-4c94-bed7-fe4459967ed4-multus-conf-dir\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622860 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-openvswitch\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-etc-tuned\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-systemd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.622951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-node-log\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623031 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-env-overrides\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625071 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623107 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-node-log\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625692 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-system-cni-dir\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.625692 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623168 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-run-systemd\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625692 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bbc40-7268-4a24-8a03-401696bbf923-host-run-ovn-kubernetes\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.625692 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.623294 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bbc40-7268-4a24-8a03-401696bbf923-ovnkube-script-lib\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.626471 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.626432 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bbc40-7268-4a24-8a03-401696bbf923-ovn-node-metrics-cert\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.630712 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.630674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnx86\" (UniqueName: \"kubernetes.io/projected/35cc59a0-b27b-4c94-bed7-fe4459967ed4-kube-api-access-mnx86\") pod \"multus-r7jnz\" (UID: \"35cc59a0-b27b-4c94-bed7-fe4459967ed4\") " pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.631439 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.631174 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9wwg\" (UniqueName: \"kubernetes.io/projected/bde5f90f-0617-41bf-b36e-8ecaadc1db67-kube-api-access-h9wwg\") pod \"iptables-alerter-ctvj2\" (UID: \"bde5f90f-0617-41bf-b36e-8ecaadc1db67\") " pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.631859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.631655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnb56\" (UniqueName: \"kubernetes.io/projected/0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a-kube-api-access-tnb56\") pod \"multus-additional-cni-plugins-n9d5h\" (UID: \"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a\") " pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.631859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.631657 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pt82\" (UniqueName: \"kubernetes.io/projected/059bbc40-7268-4a24-8a03-401696bbf923-kube-api-access-6pt82\") pod \"ovnkube-node-7pb47\" (UID: \"059bbc40-7268-4a24-8a03-401696bbf923\") " pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.723656 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-socket-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-kubernetes\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f096e16-c852-477d-bcac-de5ae629e0be-hosts-file\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723746 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-kubelet-config\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-conf\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f096e16-c852-477d-bcac-de5ae629e0be-hosts-file\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-socket-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-kubernetes\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.723859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rd7\" (UniqueName: \"kubernetes.io/projected/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-kube-api-access-q8rd7\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-var-lib-kubelet\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723853 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-kubelet-config\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.723903 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gr2m\" (UniqueName: \"kubernetes.io/projected/7f096e16-c852-477d-bcac-de5ae629e0be-kube-api-access-8gr2m\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-run\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723956 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-var-lib-kubelet\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.723996 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:05.223965481 +0000 UTC m=+3.093020211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-run\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-dbus\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.723964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74f5fa0b-75e9-4866-afea-d08a6d24e683-dbus\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f096e16-c852-477d-bcac-de5ae629e0be-tmp-dir\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:04.724344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724105 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysctl-conf\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mjj\" (UniqueName: \"kubernetes.io/projected/c475ceb4-f624-47b3-9f44-37a7d7aab228-kube-api-access-r7mjj\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-modprobe-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-host\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-serviceca\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-lib-modules\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-tmp\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6j9m\" (UniqueName: \"kubernetes.io/projected/271dce04-514f-4da5-9e92-c4acc6811b50-kube-api-access-b6j9m\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-host\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-etc-selinux\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dd58267-4611-45e1-87c2-673d21df0762-agent-certs\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-modprobe-d\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-device-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724378 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f096e16-c852-477d-bcac-de5ae629e0be-tmp-dir\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.725032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724386 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-lib-modules\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-etc-tuned\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-systemd\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-device-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-sys-fs\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysconfig\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724553 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-systemd\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-registration-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-host\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqrb\" (UniqueName: \"kubernetes.io/projected/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-kube-api-access-4jqrb\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dd58267-4611-45e1-87c2-673d21df0762-konnectivity-ca\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-sys\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-sys\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.725754 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.724791 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724811 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-sys-fs\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724841 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-host\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.724857 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:05.224841542 +0000 UTC m=+3.093896276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724793 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-serviceca\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724915 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/271dce04-514f-4da5-9e92-c4acc6811b50-etc-sysconfig\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.724933 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-registration-dir\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.725003 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c475ceb4-f624-47b3-9f44-37a7d7aab228-etc-selinux\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.726509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.725337 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dd58267-4611-45e1-87c2-673d21df0762-konnectivity-ca\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.727392 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.727362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-tmp\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.727392 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.727381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/271dce04-514f-4da5-9e92-c4acc6811b50-etc-tuned\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.727634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.727598 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dd58267-4611-45e1-87c2-673d21df0762-agent-certs\") pod \"konnectivity-agent-jcjrj\" (UID: \"6dd58267-4611-45e1-87c2-673d21df0762\") " pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.733863 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.733652 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:04.733863 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.733682 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:04.733863 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.733700 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:04.733863 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:04.733792 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:05.233768024 +0000 UTC m=+3.102822744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:04.736381 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.736308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rd7\" (UniqueName: \"kubernetes.io/projected/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-kube-api-access-q8rd7\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:04.736556 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.736450 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gr2m\" (UniqueName: \"kubernetes.io/projected/7f096e16-c852-477d-bcac-de5ae629e0be-kube-api-access-8gr2m\") pod \"node-resolver-tnqkm\" (UID: \"7f096e16-c852-477d-bcac-de5ae629e0be\") " pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:04.737091 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.737042 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqrb\" (UniqueName: \"kubernetes.io/projected/a85dfcbb-c1a1-424a-82ec-83c502cd88b3-kube-api-access-4jqrb\") pod \"node-ca-7f85s\" (UID: \"a85dfcbb-c1a1-424a-82ec-83c502cd88b3\") " pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.737091 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.737067 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mjj\" (UniqueName: \"kubernetes.io/projected/c475ceb4-f624-47b3-9f44-37a7d7aab228-kube-api-access-r7mjj\") pod \"aws-ebs-csi-driver-node-fczvf\" (UID: \"c475ceb4-f624-47b3-9f44-37a7d7aab228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.737737 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.737713 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6j9m\" (UniqueName: \"kubernetes.io/projected/271dce04-514f-4da5-9e92-c4acc6811b50-kube-api-access-b6j9m\") pod \"tuned-nwjhk\" (UID: \"271dce04-514f-4da5-9e92-c4acc6811b50\") " pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.780548 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.780483 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"78e29df1a4e6bf20e982f0f14de11bd1","Type":"ContainerStarted","Data":"f94c37d6d4d63173919e9076e63532f5218f3691ed1eabe4cf0a042222b4ebb4"} Jul 02 20:33:04.781697 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.781659 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"6394b7562ea99074d8a3f3fd7a1feb82","Type":"ContainerStarted","Data":"5057165158f2b31d7fff605d7b4f99bd0cce0cb399e4b29350d4cdcdb64a47f0"} Jul 02 20:33:04.810120 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.810081 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:04.820672 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.820638 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctvj2" Jul 02 20:33:04.839035 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.838989 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:04.845009 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.844971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7f85s" Jul 02 20:33:04.853491 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.853439 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7jnz" Jul 02 20:33:04.865998 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.865957 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" Jul 02 20:33:04.874570 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.874527 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" Jul 02 20:33:04.882129 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.882092 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" Jul 02 20:33:04.889599 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:04.889559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnqkm" Jul 02 20:33:05.022141 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.022062 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:05.226968 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.226927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:05.227153 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.227008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:05.227153 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.227109 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:05.227261 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.227168 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:06.227154589 +0000 UTC m=+4.096209315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:05.227261 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.227108 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:05.227350 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.227281 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:06.227260268 +0000 UTC m=+4.096315002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:05.327879 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.327774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:05.328033 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.327980 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:05.328033 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.328009 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:05.328033 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.328021 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:05.328183 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.328087 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:06.328067992 +0000 UTC m=+4.197122705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:05.545483 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.545421 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:28:04 +0000 UTC" deadline="2027-12-07 10:18:51.14552201 +0000 UTC" Jul 02 20:33:05.545483 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.545475 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12541h45m45.600052345s" Jul 02 20:33:05.736356 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.736317 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc475ceb4_f624_47b3_9f44_37a7d7aab228.slice/crio-ed7d28a79a65f0b323f32f10154da41789a636cba65cc7271c10047b12971a30 WatchSource:0}: Error finding container ed7d28a79a65f0b323f32f10154da41789a636cba65cc7271c10047b12971a30: Status 404 returned error can't find the container with id ed7d28a79a65f0b323f32f10154da41789a636cba65cc7271c10047b12971a30 Jul 02 20:33:05.739029 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.738999 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f096e16_c852_477d_bcac_de5ae629e0be.slice/crio-c3a7c8215adaf217993cfb2b14b0319125140130036237353e9057d6e3678996 WatchSource:0}: Error finding container c3a7c8215adaf217993cfb2b14b0319125140130036237353e9057d6e3678996: Status 404 returned error can't find the container with id c3a7c8215adaf217993cfb2b14b0319125140130036237353e9057d6e3678996 Jul 02 20:33:05.743378 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.743277 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde5f90f_0617_41bf_b36e_8ecaadc1db67.slice/crio-483159419dc66a90cb3516f0be639f887251800da5df0e7980e264d804f7bfde WatchSource:0}: Error finding container 483159419dc66a90cb3516f0be639f887251800da5df0e7980e264d804f7bfde: Status 404 returned error can't find the container with id 483159419dc66a90cb3516f0be639f887251800da5df0e7980e264d804f7bfde Jul 02 20:33:05.744129 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.744094 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85dfcbb_c1a1_424a_82ec_83c502cd88b3.slice/crio-0a0488fc724af2ff78487b22d5234d7a70031c2a3d5e820b7cf1e339e3015bd7 WatchSource:0}: Error finding container 0a0488fc724af2ff78487b22d5234d7a70031c2a3d5e820b7cf1e339e3015bd7: Status 404 returned error can't find the container with id 0a0488fc724af2ff78487b22d5234d7a70031c2a3d5e820b7cf1e339e3015bd7 Jul 02 20:33:05.746021 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.745927 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd58267_4611_45e1_87c2_673d21df0762.slice/crio-78e8cccc695fc615140c606fe81b23693e3c7722741fb3d2e60000bb53d29d87 WatchSource:0}: Error finding container 78e8cccc695fc615140c606fe81b23693e3c7722741fb3d2e60000bb53d29d87: Status 404 returned error can't find the container with id 78e8cccc695fc615140c606fe81b23693e3c7722741fb3d2e60000bb53d29d87 Jul 02 20:33:05.747484 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.747243 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7dfe8c_cdb0_4c0e_ad84_647ef1e1637a.slice/crio-0afd08fbb2ada4be8df012d8bf7f8a95a3b59c5c6cad88213207ef3fa67ac782 WatchSource:0}: Error finding container 0afd08fbb2ada4be8df012d8bf7f8a95a3b59c5c6cad88213207ef3fa67ac782: Status 404 returned error can't find the container with id 0afd08fbb2ada4be8df012d8bf7f8a95a3b59c5c6cad88213207ef3fa67ac782 Jul 02 20:33:05.748967 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.748938 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cc59a0_b27b_4c94_bed7_fe4459967ed4.slice/crio-8c457ad44ab85cf4c7b8b7272ccf231ce8d194208725c60b1bfb6f10914e646e WatchSource:0}: Error finding container 8c457ad44ab85cf4c7b8b7272ccf231ce8d194208725c60b1bfb6f10914e646e: Status 404 returned error can't find the container with id 8c457ad44ab85cf4c7b8b7272ccf231ce8d194208725c60b1bfb6f10914e646e Jul 02 20:33:05.749791 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.749764 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059bbc40_7268_4a24_8a03_401696bbf923.slice/crio-15da71d3d28438946f0b850d642d719af75c6281d77af18fa7c7c60bc209512a WatchSource:0}: Error finding container 15da71d3d28438946f0b850d642d719af75c6281d77af18fa7c7c60bc209512a: Status 404 returned error can't find the container with id 15da71d3d28438946f0b850d642d719af75c6281d77af18fa7c7c60bc209512a Jul 02 20:33:05.752436 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:05.752403 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271dce04_514f_4da5_9e92_c4acc6811b50.slice/crio-a1f6faead5374445b82509a6da33db681c4f1307f227d94bb12f8f6c08779c42 WatchSource:0}: Error finding container a1f6faead5374445b82509a6da33db681c4f1307f227d94bb12f8f6c08779c42: Status 404 returned error can't find the container with id a1f6faead5374445b82509a6da33db681c4f1307f227d94bb12f8f6c08779c42 Jul 02 20:33:05.776411 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.776380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:05.776587 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.776422 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:05.776587 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.776528 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:05.776690 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:05.776617 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:05.785031 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.784998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" event={"ID":"271dce04-514f-4da5-9e92-c4acc6811b50","Type":"ContainerStarted","Data":"a1f6faead5374445b82509a6da33db681c4f1307f227d94bb12f8f6c08779c42"} Jul 02 20:33:05.790120 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.790083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7jnz" event={"ID":"35cc59a0-b27b-4c94-bed7-fe4459967ed4","Type":"ContainerStarted","Data":"8c457ad44ab85cf4c7b8b7272ccf231ce8d194208725c60b1bfb6f10914e646e"} Jul 02 20:33:05.797470 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.797410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerStarted","Data":"0afd08fbb2ada4be8df012d8bf7f8a95a3b59c5c6cad88213207ef3fa67ac782"} Jul 02 20:33:05.801788 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.801650 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7f85s" event={"ID":"a85dfcbb-c1a1-424a-82ec-83c502cd88b3","Type":"ContainerStarted","Data":"0a0488fc724af2ff78487b22d5234d7a70031c2a3d5e820b7cf1e339e3015bd7"} Jul 02 20:33:05.802913 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.802885 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnqkm" event={"ID":"7f096e16-c852-477d-bcac-de5ae629e0be","Type":"ContainerStarted","Data":"c3a7c8215adaf217993cfb2b14b0319125140130036237353e9057d6e3678996"} Jul 02 20:33:05.804110 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.804075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"15da71d3d28438946f0b850d642d719af75c6281d77af18fa7c7c60bc209512a"} Jul 02 20:33:05.805159 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.805131 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jcjrj" event={"ID":"6dd58267-4611-45e1-87c2-673d21df0762","Type":"ContainerStarted","Data":"78e8cccc695fc615140c606fe81b23693e3c7722741fb3d2e60000bb53d29d87"} Jul 02 20:33:05.806199 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.806177 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctvj2" event={"ID":"bde5f90f-0617-41bf-b36e-8ecaadc1db67","Type":"ContainerStarted","Data":"483159419dc66a90cb3516f0be639f887251800da5df0e7980e264d804f7bfde"} Jul 02 20:33:05.807110 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:05.807088 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" event={"ID":"c475ceb4-f624-47b3-9f44-37a7d7aab228","Type":"ContainerStarted","Data":"ed7d28a79a65f0b323f32f10154da41789a636cba65cc7271c10047b12971a30"} Jul 02 20:33:06.235660 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.235341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:06.235660 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.235647 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:06.235969 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.235722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:06.235969 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.235769 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:08.235747796 +0000 UTC m=+6.104802522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:06.235969 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.235829 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:06.235969 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.235887 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:08.235868197 +0000 UTC m=+6.104922912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:06.271523 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.270901 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:33:06.337003 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.336950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:06.337215 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.337181 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:06.337215 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.337205 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:06.337321 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.337219 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:06.337321 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.337279 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:08.337260641 +0000 UTC m=+6.206315368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:06.546568 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.546462 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:28:04 +0000 UTC" deadline="2028-01-23 10:47:39.293940048 +0000 UTC" Jul 02 20:33:06.546568 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.546533 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13670h14m32.747411227s" Jul 02 20:33:06.777259 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.776681 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:06.777259 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:06.776815 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:06.846078 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.845163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"78e29df1a4e6bf20e982f0f14de11bd1","Type":"ContainerStarted","Data":"52741a4e0064a281876a2e9fd2fbcdc3dab4db3128cd1386440b8cd71f92e343"} Jul 02 20:33:06.860783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.860717 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" podStartSLOduration=3.8606943559999998 podStartE2EDuration="3.860694356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:06.859710824 +0000 UTC m=+4.728765560" watchObservedRunningTime="2026-07-02 20:33:06.860694356 +0000 UTC m=+4.729749076" Jul 02 20:33:06.861182 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.861152 2569 generic.go:356] "Generic (PLEG): container finished" podID="6394b7562ea99074d8a3f3fd7a1feb82" containerID="5bfe2b45c927c3586a40b9fc797a15ef98cf55660c6b376f6d59d79e2d659631" exitCode=0 Jul 02 20:33:06.861279 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:06.861206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"6394b7562ea99074d8a3f3fd7a1feb82","Type":"ContainerDied","Data":"5bfe2b45c927c3586a40b9fc797a15ef98cf55660c6b376f6d59d79e2d659631"} Jul 02 20:33:07.777081 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:07.776491 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:07.777081 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:07.776515 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:07.777081 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:07.776620 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:07.777081 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:07.777042 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:07.882611 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:07.882505 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"6394b7562ea99074d8a3f3fd7a1feb82","Type":"ContainerStarted","Data":"7858c289973e1f26d178353e7f3b389166d905481b362368cf0695a41b187e57"} Jul 02 20:33:08.259474 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:08.259384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:08.259634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:08.259471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:08.259634 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.259604 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:08.259736 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.259658 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:12.259640747 +0000 UTC m=+10.128695464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:08.260072 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.260043 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:08.260159 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.260097 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:12.260081962 +0000 UTC m=+10.129136682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:08.359758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:08.359721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:08.360050 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.359976 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:08.360050 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.359999 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:08.360050 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.360011 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:08.360240 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.360066 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:12.36004926 +0000 UTC m=+10.229103974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:08.776631 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:08.776606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:08.776784 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:08.776725 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:09.775935 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:09.775902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:09.776387 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:09.776036 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:09.776387 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:09.776095 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:09.776387 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:09.776220 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:10.776543 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:10.776135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:10.776543 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:10.776262 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:11.775665 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:11.775635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:11.775840 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:11.775681 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:11.775840 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:11.775794 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:11.775956 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:11.775932 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:12.288159 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:12.288126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:12.288626 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:12.288190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:12.288626 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.288287 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:12.288626 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.288314 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:12.288626 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.288353 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:20.288334207 +0000 UTC m=+18.157388929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:12.288626 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.288368 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:20.288361576 +0000 UTC m=+18.157416290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:12.389233 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:12.389134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:12.389398 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.389288 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:12.389398 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.389305 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:12.389398 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.389319 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:12.389398 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.389381 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:20.389363309 +0000 UTC m=+18.258418027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:12.777519 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:12.777412 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:12.777688 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:12.777550 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:13.776471 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:13.776019 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:13.776471 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:13.776128 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:13.776471 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:13.776345 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:13.777004 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:13.776486 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:14.775822 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:14.775790 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:14.776023 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:14.775912 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:15.776520 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:15.776485 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:15.776971 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:15.776539 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:15.776971 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:15.776651 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:15.776971 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:15.776744 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:16.775863 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:16.775830 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:16.776039 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:16.775955 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:17.776312 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:17.776287 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:17.776859 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:17.776416 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:17.776859 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:17.776476 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:17.776859 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:17.776573 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:18.776527 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:18.776495 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:18.776967 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:18.776616 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:19.775651 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:19.775621 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:19.775813 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:19.775621 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:19.775813 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:19.775712 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:19.775876 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:19.775836 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:20.348310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:20.348277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:20.348767 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:20.348336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:20.348767 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.348434 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:20.348767 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.348434 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:20.348767 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.348500 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:36.348484287 +0000 UTC m=+34.217539001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:20.348767 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.348532 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:36.348510212 +0000 UTC m=+34.217564926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:20.449434 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:20.449406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:20.449604 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.449572 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:20.449604 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.449597 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:20.449604 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.449607 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:20.449766 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.449654 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:36.449640613 +0000 UTC m=+34.318695340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:20.776593 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:20.776506 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:20.776736 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:20.776657 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:21.775783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:21.775755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:21.775783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:21.775791 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:21.776278 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:21.775891 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:21.776278 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:21.776038 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:22.776772 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:22.776741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:22.777134 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:22.776855 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:23.775746 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:23.775678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:23.775746 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:23.775741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:23.775966 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:23.775832 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:23.776017 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:23.775969 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:24.776600 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.776468 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:24.777149 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:24.776717 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:24.916114 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.916090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" event={"ID":"271dce04-514f-4da5-9e92-c4acc6811b50","Type":"ContainerStarted","Data":"677cd6f357c820e27487cfb9fd508b788e843092ae7667802fd7f5dae866513a"} Jul 02 20:33:24.917589 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.917560 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7jnz" event={"ID":"35cc59a0-b27b-4c94-bed7-fe4459967ed4","Type":"ContainerStarted","Data":"abb83e889fbc9c8d18c0a8fe65fdcfae05f53d8de6b9f96f45b97e4f44bee342"} Jul 02 20:33:24.919263 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.919237 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="dfedfd0f9a9880d43b92b1335afdc9735c3bbea198a9707b6a2147302bc0aa1a" exitCode=0 Jul 02 20:33:24.919368 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.919305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"dfedfd0f9a9880d43b92b1335afdc9735c3bbea198a9707b6a2147302bc0aa1a"} Jul 02 20:33:24.920976 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.920955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7f85s" event={"ID":"a85dfcbb-c1a1-424a-82ec-83c502cd88b3","Type":"ContainerStarted","Data":"ea2b0f3bee2b26c1b4205db18cf27c2be424c0880e682590dd8027a81df2c8f5"} Jul 02 20:33:24.925062 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.925037 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnqkm" event={"ID":"7f096e16-c852-477d-bcac-de5ae629e0be","Type":"ContainerStarted","Data":"c11ec51da3d89dfc509e1ff35f2e3249cc52a17701daab08f50921700d659125"} Jul 02 20:33:24.928592 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.928565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"af1affc68e3876a02ee9125a5660b5431d9692775702b3affb8b87f49c744c52"} Jul 02 20:33:24.928687 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.928646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"c301feba1ca1b4a6c14c370f50e7d3f3317202eee74d526b8ac8f8c778c05d9b"} Jul 02 20:33:24.928758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.928699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"686857c01e79a76e9a200f31868d4683f8035f62ddb0c4515554124dec17c64f"} Jul 02 20:33:24.928758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.928745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"c781713f7a546884e724af5cd7b6e301c9b74e1adb60dc4c5d4429d99cf3efbb"} Jul 02 20:33:24.928900 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.928797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"2d9f473ff4d794d0da9d057c73db5400b570f96665ee95346800a9a7d2595506"} Jul 02 20:33:24.930006 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.929982 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jcjrj" event={"ID":"6dd58267-4611-45e1-87c2-673d21df0762","Type":"ContainerStarted","Data":"a2a08806886322fa7946b7509d12171783dc667527cf6c05a862b405cb14f29a"} Jul 02 20:33:24.931592 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.931568 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" event={"ID":"c475ceb4-f624-47b3-9f44-37a7d7aab228","Type":"ContainerStarted","Data":"4babea97eb268c216aaaf9d8eae231a846109933cd581ee7f742a2dd4d05b208"} Jul 02 20:33:24.941364 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.941327 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" podStartSLOduration=21.941316683 podStartE2EDuration="21.941316683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:07.904633516 +0000 UTC m=+5.773688253" watchObservedRunningTime="2026-07-02 20:33:24.941316683 +0000 UTC m=+22.810371418" Jul 02 20:33:24.941813 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.941788 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nwjhk" podStartSLOduration=4.399915202 podStartE2EDuration="22.941782871s" totalImagesPullingTime="18.541867669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:24.94115325 +0000 UTC m=+22.810207986" watchObservedRunningTime="2026-07-02 20:33:24.941782871 +0000 UTC m=+22.810837606" Jul 02 20:33:24.958340 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.958292 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r7jnz" podStartSLOduration=4.020799833 podStartE2EDuration="22.958274928s" totalImagesPullingTime="18.937475095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:24.957852933 +0000 UTC m=+22.826907669" watchObservedRunningTime="2026-07-02 20:33:24.958274928 +0000 UTC m=+22.827329664" Jul 02 20:33:24.973072 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.973035 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tnqkm" podStartSLOduration=4.412439706 podStartE2EDuration="22.973023679s" totalImagesPullingTime="18.560583973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:24.972907834 +0000 UTC m=+22.841962571" watchObservedRunningTime="2026-07-02 20:33:24.973023679 +0000 UTC m=+22.842078413" Jul 02 20:33:24.988424 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:24.988380 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7f85s" podStartSLOduration=4.449653844 podStartE2EDuration="22.988363683s" totalImagesPullingTime="18.538709839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:24.988241086 +0000 UTC m=+22.857295823" watchObservedRunningTime="2026-07-02 20:33:24.988363683 +0000 UTC m=+22.857418419" Jul 02 20:33:25.003147 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.003101 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jcjrj" podStartSLOduration=4.454267478 podStartE2EDuration="23.003090738s" totalImagesPullingTime="18.54882326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:25.002599219 +0000 UTC m=+22.871653951" watchObservedRunningTime="2026-07-02 20:33:25.003090738 +0000 UTC m=+22.872145472" Jul 02 20:33:25.752368 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.752345 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 20:33:25.775867 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.775834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:25.775972 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:25.775941 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:25.776037 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.776003 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:25.776138 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:25.776109 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:25.937672 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.937612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"ab966df183291df200bfd9aff76d07e638c00c1c5ac2dffb864d75e394b69bae"} Jul 02 20:33:25.939306 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.939274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctvj2" event={"ID":"bde5f90f-0617-41bf-b36e-8ecaadc1db67","Type":"ContainerStarted","Data":"3f829ec43b37a45c6aaa57d8296667077f1d4a199b2843c89e79f79150516c69"} Jul 02 20:33:25.941361 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.941308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" event={"ID":"c475ceb4-f624-47b3-9f44-37a7d7aab228","Type":"ContainerStarted","Data":"42087c7c1956e037790abc8cc11e42d3c8cef5508b78632aded76929a3f818dd"} Jul 02 20:33:25.954645 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:25.954601 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ctvj2" podStartSLOduration=5.415454177 podStartE2EDuration="23.954587622s" totalImagesPullingTime="18.539133445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:25.953946308 +0000 UTC m=+23.823001045" watchObservedRunningTime="2026-07-02 20:33:25.954587622 +0000 UTC m=+23.823642351" Jul 02 20:33:26.630257 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:26.630160 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T20:33:25.752365067Z","UUID":"3045bb14-9b10-4918-9f61-ccf3d622cb6d","Handler":null,"Name":"","Endpoint":""} Jul 02 20:33:26.631525 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:26.631508 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 20:33:26.631615 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:26.631541 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 20:33:26.775998 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:26.775973 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:26.776125 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:26.776085 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:27.167029 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.167007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:27.167585 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.167567 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:27.184013 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.183994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:27.184899 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.184882 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jcjrj" Jul 02 20:33:27.776158 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.775948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:27.776322 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.775992 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:27.776322 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:27.776266 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:27.776443 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:27.776326 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:27.949905 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.949867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"81b475387d71c4438daf27b1dbd08f83903f5d6bb0e63fb0028f23416c45086b"} Jul 02 20:33:27.952294 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.952249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" event={"ID":"c475ceb4-f624-47b3-9f44-37a7d7aab228","Type":"ContainerStarted","Data":"d3674572e6f01e200eb3a2541298c8daf2568f23a9c2a129f87a193961824698"} Jul 02 20:33:27.972432 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:27.972387 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fczvf" podStartSLOduration=5.126794438 podStartE2EDuration="25.972373726s" totalImagesPullingTime="20.845579288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:27.97182703 +0000 UTC m=+25.840881767" watchObservedRunningTime="2026-07-02 20:33:27.972373726 +0000 UTC m=+25.841428461" Jul 02 20:33:28.775664 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:28.775636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:28.776160 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:28.775749 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:29.776425 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.776401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:29.776833 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:29.776510 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:29.776833 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.776571 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:29.776833 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:29.776650 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:29.960032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.959860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" event={"ID":"059bbc40-7268-4a24-8a03-401696bbf923","Type":"ContainerStarted","Data":"d2502bf7211aa858fa0efc6d90fc446d6c919ff8dfbf04bd58f556220010e381"} Jul 02 20:33:29.960183 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.960096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:29.974872 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.974850 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:29.987968 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:29.987929 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" podStartSLOduration=9.149444094 podStartE2EDuration="27.987917032s" totalImagesPullingTime="18.838472938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:29.987307565 +0000 UTC m=+27.856362300" watchObservedRunningTime="2026-07-02 20:33:29.987917032 +0000 UTC m=+27.856971766" Jul 02 20:33:30.776081 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.776020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:30.776209 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:30.776144 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:30.963728 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.963695 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="20a5c132cf09b1ba19e5d9628a07a0bdf3cec2814d879e2a31bdccedfe7c1d01" exitCode=0 Jul 02 20:33:30.964108 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.963754 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"20a5c132cf09b1ba19e5d9628a07a0bdf3cec2814d879e2a31bdccedfe7c1d01"} Jul 02 20:33:30.965112 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.964415 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:30.965112 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.964500 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:30.987205 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:30.987182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:33:31.775841 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.775809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:31.775966 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.775845 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:31.775966 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:31.775947 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:31.776107 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:31.776085 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:31.994256 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.994123 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdqwm"] Jul 02 20:33:31.994583 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.994330 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:31.994583 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:31.994411 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:31.999438 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.999375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn2fv"] Jul 02 20:33:31.999647 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.999491 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:31.999647 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:31.999598 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:31.999987 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:31.999968 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5ssn"] Jul 02 20:33:32.000047 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:32.000036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:32.000133 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:32.000111 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:32.972181 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:32.972142 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="a6a9e9185023586da9a15a2e5917b65a8f2afc525677148ef1934def2e764eb6" exitCode=0 Jul 02 20:33:32.973671 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:32.973630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"a6a9e9185023586da9a15a2e5917b65a8f2afc525677148ef1934def2e764eb6"} Jul 02 20:33:33.776219 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:33.776138 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:33.776219 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:33.776181 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:33.776219 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:33.776181 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:33.776686 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:33.776258 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:33.776686 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:33.776328 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:33.776686 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:33.776389 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:34.979534 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:34.979502 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="f0a6981b75ceee176063f9a6fa539cc79549f3cf9abca09ed689141acae8e778" exitCode=0 Jul 02 20:33:34.980084 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:34.979546 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"f0a6981b75ceee176063f9a6fa539cc79549f3cf9abca09ed689141acae8e778"} Jul 02 20:33:35.776323 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:35.776290 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:35.776512 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:35.776405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:35.776512 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:35.776414 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:35.776512 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:35.776431 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:35.776711 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:35.776545 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:35.776711 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:35.776632 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:36.378813 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:36.378742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:36.378813 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:36.378788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:36.379300 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.378881 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:36.379300 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.378892 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:36.379300 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.378932 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret podName:74f5fa0b-75e9-4866-afea-d08a6d24e683 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:08.37891666 +0000 UTC m=+66.247971392 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret") pod "global-pull-secret-syncer-qdqwm" (UID: "74f5fa0b-75e9-4866-afea-d08a6d24e683") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:33:36.379300 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.378958 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:08.378937838 +0000 UTC m=+66.247992568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:33:36.479644 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:36.479618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:36.479824 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.479800 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:33:36.479959 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.479835 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:33:36.479959 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.479849 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vcxnr for pod openshift-network-diagnostics/network-check-target-w5ssn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:36.479959 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:36.479903 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr podName:8aa26210-1768-4a93-91ba-ac9bf7ea6318 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:08.479887158 +0000 UTC m=+66.348941888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcxnr" (UniqueName: "kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr") pod "network-check-target-w5ssn" (UID: "8aa26210-1768-4a93-91ba-ac9bf7ea6318") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:33:37.775646 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:37.775618 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:37.776096 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:37.775660 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:37.776096 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:37.775738 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:37.776096 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:37.775748 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn2fv" podUID="a7ca8736-448b-4aeb-bd02-d42cd0f778f4" Jul 02 20:33:37.776096 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:37.775861 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdqwm" podUID="74f5fa0b-75e9-4866-afea-d08a6d24e683" Jul 02 20:33:37.776096 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:37.775947 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w5ssn" podUID="8aa26210-1768-4a93-91ba-ac9bf7ea6318" Jul 02 20:33:38.383362 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.383333 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeReady" Jul 02 20:33:38.383540 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.383479 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 20:33:38.445012 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.444934 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4k6j5"] Jul 02 20:33:38.448296 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.448269 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lk8nz"] Jul 02 20:33:38.448423 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.448369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.450155 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.450134 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.451192 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.451172 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 20:33:38.451334 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.451250 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 20:33:38.451498 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.451432 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h2bvt\"" Jul 02 20:33:38.451674 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.451657 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 20:33:38.452712 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.452692 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 20:33:38.453081 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.453056 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 20:33:38.453183 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.453090 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w7dxd\"" Jul 02 20:33:38.457235 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.457178 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4k6j5"] Jul 02 20:33:38.460029 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.459976 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lk8nz"] Jul 02 20:33:38.596646 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.596785 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac5aff-ebdd-415c-8b7d-46af46993d65-config-volume\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.596785 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596694 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751d0909-edfc-4b48-9019-c83d5baea0cb-cert\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.596785 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9sfg\" (UniqueName: \"kubernetes.io/projected/36ac5aff-ebdd-415c-8b7d-46af46993d65-kube-api-access-n9sfg\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.596785 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596765 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4hn\" (UniqueName: \"kubernetes.io/projected/751d0909-edfc-4b48-9019-c83d5baea0cb-kube-api-access-hj4hn\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.596950 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.596824 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ac5aff-ebdd-415c-8b7d-46af46993d65-tmp-dir\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.697987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ac5aff-ebdd-415c-8b7d-46af46993d65-tmp-dir\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698180 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698180 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac5aff-ebdd-415c-8b7d-46af46993d65-config-volume\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698180 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:38.698151 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:38.698330 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698184 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751d0909-edfc-4b48-9019-c83d5baea0cb-cert\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.698330 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:38.698198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:39.198184271 +0000 UTC m=+37.067238999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:38.698330 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9sfg\" (UniqueName: \"kubernetes.io/projected/36ac5aff-ebdd-415c-8b7d-46af46993d65-kube-api-access-n9sfg\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698330 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4hn\" (UniqueName: \"kubernetes.io/projected/751d0909-edfc-4b48-9019-c83d5baea0cb-kube-api-access-hj4hn\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.698330 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ac5aff-ebdd-415c-8b7d-46af46993d65-tmp-dir\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.698793 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.698772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ac5aff-ebdd-415c-8b7d-46af46993d65-config-volume\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.703002 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.702835 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/751d0909-edfc-4b48-9019-c83d5baea0cb-cert\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.707231 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.707187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9sfg\" (UniqueName: \"kubernetes.io/projected/36ac5aff-ebdd-415c-8b7d-46af46993d65-kube-api-access-n9sfg\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:38.707231 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.707217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4hn\" (UniqueName: \"kubernetes.io/projected/751d0909-edfc-4b48-9019-c83d5baea0cb-kube-api-access-hj4hn\") pod \"ingress-canary-4k6j5\" (UID: \"751d0909-edfc-4b48-9019-c83d5baea0cb\") " pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:38.760970 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:38.760947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4k6j5" Jul 02 20:33:39.201849 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.201819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:39.202134 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:39.201932 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:39.202134 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:39.202014 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:40.201991577 +0000 UTC m=+38.071046291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:39.270639 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.270616 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4k6j5"] Jul 02 20:33:39.275680 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:39.275647 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751d0909_edfc_4b48_9019_c83d5baea0cb.slice/crio-fbd8a39b85e9bfe42a6a1116702f6eccd0e0fbf82ba26a2d7770733a2f5a3810 WatchSource:0}: Error finding container fbd8a39b85e9bfe42a6a1116702f6eccd0e0fbf82ba26a2d7770733a2f5a3810: Status 404 returned error can't find the container with id fbd8a39b85e9bfe42a6a1116702f6eccd0e0fbf82ba26a2d7770733a2f5a3810 Jul 02 20:33:39.776080 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.776052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:33:39.776080 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.776074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:33:39.776270 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.776074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:33:39.778908 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.778886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:33:39.780310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.780202 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 20:33:39.780310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.780211 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 20:33:39.780310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.780236 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:33:39.780310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.780240 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zddh5\"" Jul 02 20:33:39.780310 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.780258 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w2d5j\"" Jul 02 20:33:39.992570 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.992508 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="1283f522f2fbefa8f25bb13fbdacdc779abd1644cf1e3ef5b54c5324b71043fd" exitCode=0 Jul 02 20:33:39.992680 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.992569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"1283f522f2fbefa8f25bb13fbdacdc779abd1644cf1e3ef5b54c5324b71043fd"} Jul 02 20:33:39.993621 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:39.993604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4k6j5" event={"ID":"751d0909-edfc-4b48-9019-c83d5baea0cb","Type":"ContainerStarted","Data":"fbd8a39b85e9bfe42a6a1116702f6eccd0e0fbf82ba26a2d7770733a2f5a3810"} Jul 02 20:33:40.207797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:40.207766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:40.208100 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:40.207950 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:40.208100 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:40.208024 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:42.208006352 +0000 UTC m=+40.077061067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:40.999242 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:40.999209 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a" containerID="6d1c874f94be3aa5421e37dceeb5eb616e064b37a8fe820babcc8893a0b622a5" exitCode=0 Jul 02 20:33:40.999414 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:40.999259 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerDied","Data":"6d1c874f94be3aa5421e37dceeb5eb616e064b37a8fe820babcc8893a0b622a5"} Jul 02 20:33:42.005784 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:42.005750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" event={"ID":"0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a","Type":"ContainerStarted","Data":"f4e1e81062d8b2fdec4e5d78dc7a9a17d6fe40fae9271a7ffcff3c0c5aad624e"} Jul 02 20:33:42.007476 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:42.007434 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4k6j5" event={"ID":"751d0909-edfc-4b48-9019-c83d5baea0cb","Type":"ContainerStarted","Data":"32fa92c721271f8ac1c7455d04a0d025b032dd21745ad2d83759d5fb8c026dc5"} Jul 02 20:33:42.029787 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:42.029733 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n9d5h" podStartSLOduration=10.196458703 podStartE2EDuration="40.029714458s" totalImagesPullingTime="29.833255755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:42.028155484 +0000 UTC m=+39.897210220" watchObservedRunningTime="2026-07-02 20:33:42.029714458 +0000 UTC m=+39.898769194" Jul 02 20:33:42.046601 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:42.046552 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4k6j5" podStartSLOduration=1.46587058 podStartE2EDuration="4.046533931s" totalImagesPullingTime="2.580663351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:42.045626655 +0000 UTC m=+39.914681392" watchObservedRunningTime="2026-07-02 20:33:42.046533931 +0000 UTC m=+39.915588667" Jul 02 20:33:42.223650 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:42.223572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:42.223769 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:42.223693 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:42.223769 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:42.223745 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:46.223732164 +0000 UTC m=+44.092786878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:46.249942 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:46.249901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:46.250481 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:46.250037 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:46.250481 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:46.250114 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:54.250093581 +0000 UTC m=+52.119148360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:54.301367 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.301328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:33:54.301885 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:54.301508 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:33:54.301885 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:54.301582 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls podName:36ac5aff-ebdd-415c-8b7d-46af46993d65 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:10.301565404 +0000 UTC m=+68.170620118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls") pod "dns-default-lk8nz" (UID: "36ac5aff-ebdd-415c-8b7d-46af46993d65") : secret "dns-default-metrics-tls" not found Jul 02 20:33:54.795777 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.795751 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f"] Jul 02 20:33:54.809141 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.809117 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f"] Jul 02 20:33:54.809273 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.809204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:54.813359 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.813343 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 20:33:54.814826 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.814802 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 02 20:33:54.815036 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.814816 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 20:33:54.815145 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.814832 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 02 20:33:54.815145 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.814912 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tlg6n\"" Jul 02 20:33:54.905196 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.905174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/907b49ba-ac93-4909-8998-671f55aa7697-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:54.905282 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.905209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:54.905282 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.905249 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwqv\" (UniqueName: \"kubernetes.io/projected/907b49ba-ac93-4909-8998-671f55aa7697-kube-api-access-fvwqv\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:54.994842 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:54.994820 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg"] Jul 02 20:33:55.006336 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.006310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwqv\" (UniqueName: \"kubernetes.io/projected/907b49ba-ac93-4909-8998-671f55aa7697-kube-api-access-fvwqv\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.006419 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.006366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/907b49ba-ac93-4909-8998-671f55aa7697-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.006419 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.006390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.006525 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.006489 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:55.006558 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.006549 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:55.506528349 +0000 UTC m=+53.375583062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:55.007041 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.007024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/907b49ba-ac93-4909-8998-671f55aa7697-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.012806 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.012780 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:33:55.012925 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.012909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" Jul 02 20:33:55.018797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.018759 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:33:55.018797 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.018760 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 02 20:33:55.019051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.019034 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cl97m\"" Jul 02 20:33:55.024625 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.024605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwqv\" (UniqueName: \"kubernetes.io/projected/907b49ba-ac93-4909-8998-671f55aa7697-kube-api-access-fvwqv\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.036723 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.036700 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.036804 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.036644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg"] Jul 02 20:33:55.040977 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.040960 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 20:33:55.041070 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.040985 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 20:33:55.041070 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.041003 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 20:33:55.041070 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.041037 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kxfdk\"" Jul 02 20:33:55.045331 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.045314 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:33:55.052251 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.052195 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 20:33:55.106747 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.106846 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.106846 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.106846 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.106846 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.107051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.107051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grrb\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.107051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.106972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.107051 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.107015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6w5\" (UniqueName: \"kubernetes.io/projected/346c2a3b-e613-41b7-addd-b62d91c030ea-kube-api-access-hw6w5\") pod \"volume-data-source-validator-5d456fdbcd-d6mlg\" (UID: \"346c2a3b-e613-41b7-addd-b62d91c030ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" Jul 02 20:33:55.208151 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208241 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208299 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208358 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208358 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208446 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.208413 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:33:55.208446 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.208432 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f75b44cf7-kppff: secret "image-registry-tls" not found Jul 02 20:33:55.208554 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.208506 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls podName:8102a931-a235-47f9-82b2-7b5d1d5d1cbc nodeName:}" failed. No retries permitted until 2026-07-02 20:33:55.708486588 +0000 UTC m=+53.577541321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls") pod "image-registry-7f75b44cf7-kppff" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc") : secret "image-registry-tls" not found Jul 02 20:33:55.208554 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208664 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grrb\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208717 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.208769 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.208727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6w5\" (UniqueName: \"kubernetes.io/projected/346c2a3b-e613-41b7-addd-b62d91c030ea-kube-api-access-hw6w5\") pod \"volume-data-source-validator-5d456fdbcd-d6mlg\" (UID: \"346c2a3b-e613-41b7-addd-b62d91c030ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" Jul 02 20:33:55.209039 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.209019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.209107 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.209045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.209214 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.209198 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.210947 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.210924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.211025 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.210963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.217067 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.217043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6w5\" (UniqueName: \"kubernetes.io/projected/346c2a3b-e613-41b7-addd-b62d91c030ea-kube-api-access-hw6w5\") pod \"volume-data-source-validator-5d456fdbcd-d6mlg\" (UID: \"346c2a3b-e613-41b7-addd-b62d91c030ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" Jul 02 20:33:55.217317 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.217296 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.217509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.217445 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grrb\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.321037 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.320990 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" Jul 02 20:33:55.469164 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.469137 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg"] Jul 02 20:33:55.472247 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:55.472224 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346c2a3b_e613_41b7_addd_b62d91c030ea.slice/crio-1e446ba7877dea61e1766fe809718acf0fbc93a4c586729d3117381edc7a0953 WatchSource:0}: Error finding container 1e446ba7877dea61e1766fe809718acf0fbc93a4c586729d3117381edc7a0953: Status 404 returned error can't find the container with id 1e446ba7877dea61e1766fe809718acf0fbc93a4c586729d3117381edc7a0953 Jul 02 20:33:55.510794 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.510768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:55.510881 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.510867 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:55.510934 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.510925 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:56.510909387 +0000 UTC m=+54.379964103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:55.712487 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:55.712418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:55.712615 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.712555 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:33:55.712615 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.712571 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f75b44cf7-kppff: secret "image-registry-tls" not found Jul 02 20:33:55.712615 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:55.712615 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls podName:8102a931-a235-47f9-82b2-7b5d1d5d1cbc nodeName:}" failed. No retries permitted until 2026-07-02 20:33:56.712601552 +0000 UTC m=+54.581656265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls") pod "image-registry-7f75b44cf7-kppff" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc") : secret "image-registry-tls" not found Jul 02 20:33:56.039637 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:56.039604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" event={"ID":"346c2a3b-e613-41b7-addd-b62d91c030ea","Type":"ContainerStarted","Data":"1e446ba7877dea61e1766fe809718acf0fbc93a4c586729d3117381edc7a0953"} Jul 02 20:33:56.519161 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:56.519085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:56.519562 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:56.519244 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:56.519562 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:56.519317 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:33:58.519302344 +0000 UTC m=+56.388357057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:56.720544 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:56.720510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:56.720681 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:56.720622 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:33:56.720681 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:56.720634 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f75b44cf7-kppff: secret "image-registry-tls" not found Jul 02 20:33:56.720681 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:56.720681 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls podName:8102a931-a235-47f9-82b2-7b5d1d5d1cbc nodeName:}" failed. No retries permitted until 2026-07-02 20:33:58.720667377 +0000 UTC m=+56.589722091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls") pod "image-registry-7f75b44cf7-kppff" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc") : secret "image-registry-tls" not found Jul 02 20:33:58.047393 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:58.047346 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" event={"ID":"346c2a3b-e613-41b7-addd-b62d91c030ea","Type":"ContainerStarted","Data":"c5bfab0261c1b15d35e9d0317ebb01d580ef1b576981169f4148f3818333f966"} Jul 02 20:33:58.063021 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:58.062965 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-d6mlg" podStartSLOduration=2.561279708 podStartE2EDuration="4.062949746s" totalImagesPullingTime="1.501670038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:33:58.061587536 +0000 UTC m=+55.930642271" watchObservedRunningTime="2026-07-02 20:33:58.062949746 +0000 UTC m=+55.932004480" Jul 02 20:33:58.533939 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:58.533913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:33:58.534103 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:58.534059 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:58.534153 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:58.534123 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:02.534109065 +0000 UTC m=+60.403163779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:33:58.735988 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:58.735963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:33:58.736123 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:58.736103 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:33:58.736165 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:58.736124 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f75b44cf7-kppff: secret "image-registry-tls" not found Jul 02 20:33:58.736197 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:33:58.736171 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls podName:8102a931-a235-47f9-82b2-7b5d1d5d1cbc nodeName:}" failed. No retries permitted until 2026-07-02 20:34:02.736158214 +0000 UTC m=+60.605212928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls") pod "image-registry-7f75b44cf7-kppff" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc") : secret "image-registry-tls" not found Jul 02 20:33:59.066834 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.066807 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht"] Jul 02 20:33:59.068779 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.068761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" Jul 02 20:33:59.071347 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.071326 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vlqlx\"" Jul 02 20:33:59.078724 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.078700 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht"] Jul 02 20:33:59.139662 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.139636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb94\" (UniqueName: \"kubernetes.io/projected/b6f76bd4-923a-4861-86fe-f9846254277c-kube-api-access-2kb94\") pod \"network-check-source-989f8b8dc-qdsht\" (UID: \"b6f76bd4-923a-4861-86fe-f9846254277c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" Jul 02 20:33:59.240943 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.240912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb94\" (UniqueName: \"kubernetes.io/projected/b6f76bd4-923a-4861-86fe-f9846254277c-kube-api-access-2kb94\") pod \"network-check-source-989f8b8dc-qdsht\" (UID: \"b6f76bd4-923a-4861-86fe-f9846254277c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" Jul 02 20:33:59.251399 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.251367 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb94\" (UniqueName: \"kubernetes.io/projected/b6f76bd4-923a-4861-86fe-f9846254277c-kube-api-access-2kb94\") pod \"network-check-source-989f8b8dc-qdsht\" (UID: \"b6f76bd4-923a-4861-86fe-f9846254277c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" Jul 02 20:33:59.377547 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.377486 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" Jul 02 20:33:59.517626 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:33:59.517590 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht"] Jul 02 20:33:59.520083 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:33:59.520044 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f76bd4_923a_4861_86fe_f9846254277c.slice/crio-c41921cf8072055db1caf5133e9a9d034cd82d33a367d1f243633757ce0fb04b WatchSource:0}: Error finding container c41921cf8072055db1caf5133e9a9d034cd82d33a367d1f243633757ce0fb04b: Status 404 returned error can't find the container with id c41921cf8072055db1caf5133e9a9d034cd82d33a367d1f243633757ce0fb04b Jul 02 20:34:00.052964 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.052932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" event={"ID":"b6f76bd4-923a-4861-86fe-f9846254277c","Type":"ContainerStarted","Data":"c41921cf8072055db1caf5133e9a9d034cd82d33a367d1f243633757ce0fb04b"} Jul 02 20:34:00.560882 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.560854 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9g7jf"] Jul 02 20:34:00.563173 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.563153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.566025 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.566003 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 20:34:00.566355 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.566306 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 20:34:00.566467 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.566375 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 20:34:00.566467 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.566431 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 20:34:00.567578 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.567559 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6z6g\"" Jul 02 20:34:00.578337 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.578317 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9g7jf"] Jul 02 20:34:00.652798 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.652775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.652887 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.652810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-crio-socket\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.652887 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.652845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-data-volume\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.652887 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.652869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.652990 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.652963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdxj\" (UniqueName: \"kubernetes.io/projected/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-api-access-hvdxj\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753213 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdxj\" (UniqueName: \"kubernetes.io/projected/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-api-access-hvdxj\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753313 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753313 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-crio-socket\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753417 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-data-volume\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753417 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753340 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753546 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:00.753416 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:34:00.753546 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-crio-socket\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.753546 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:00.753500 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:01.253478312 +0000 UTC m=+59.122533027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : secret "insights-runtime-extractor-tls" not found Jul 02 20:34:00.753546 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:00.753501 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:34:00.753713 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:00.753577 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:01.253558313 +0000 UTC m=+59.122613044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : configmap "kube-rbac-proxy" not found Jul 02 20:34:00.753713 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.753682 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-data-volume\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:00.761659 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:00.761638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdxj\" (UniqueName: \"kubernetes.io/projected/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-api-access-hvdxj\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:01.003732 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.003672 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m"] Jul 02 20:34:01.005902 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.005881 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" Jul 02 20:34:01.008590 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.008517 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 02 20:34:01.008713 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.008593 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gj4r\"" Jul 02 20:34:01.008713 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.008599 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 02 20:34:01.014387 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.014365 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m"] Jul 02 20:34:01.055117 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.055091 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prt8b\" (UniqueName: \"kubernetes.io/projected/9aafc3ae-4262-4207-974f-527fc8915031-kube-api-access-prt8b\") pod \"migrator-d57d58cc9-mcm2m\" (UID: \"9aafc3ae-4262-4207-974f-527fc8915031\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" Jul 02 20:34:01.155541 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.155515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt8b\" (UniqueName: \"kubernetes.io/projected/9aafc3ae-4262-4207-974f-527fc8915031-kube-api-access-prt8b\") pod \"migrator-d57d58cc9-mcm2m\" (UID: \"9aafc3ae-4262-4207-974f-527fc8915031\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" Jul 02 20:34:01.164723 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.164699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prt8b\" (UniqueName: \"kubernetes.io/projected/9aafc3ae-4262-4207-974f-527fc8915031-kube-api-access-prt8b\") pod \"migrator-d57d58cc9-mcm2m\" (UID: \"9aafc3ae-4262-4207-974f-527fc8915031\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" Jul 02 20:34:01.256943 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.256880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:01.257098 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.256944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:01.257098 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:01.257055 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:34:01.257216 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:01.257102 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:02.257090517 +0000 UTC m=+60.126145231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : configmap "kube-rbac-proxy" not found Jul 02 20:34:01.257216 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:01.257054 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:34:01.257216 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:01.257194 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:02.257176985 +0000 UTC m=+60.126231698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : secret "insights-runtime-extractor-tls" not found Jul 02 20:34:01.315802 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:01.315771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" Jul 02 20:34:02.264032 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.264009 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:02.264414 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.264055 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:02.264414 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.264141 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:34:02.264414 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.264173 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:34:02.264414 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.264185 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:04.264171448 +0000 UTC m=+62.133226164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : secret "insights-runtime-extractor-tls" not found Jul 02 20:34:02.264414 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.264251 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:04.264233298 +0000 UTC m=+62.133288011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : configmap "kube-rbac-proxy" not found Jul 02 20:34:02.342734 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.342678 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m"] Jul 02 20:34:02.346391 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:02.346366 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aafc3ae_4262_4207_974f_527fc8915031.slice/crio-6d54dada1b1df7462c6b074cc47ef71dfc06691a229855ea425da49f9f842df5 WatchSource:0}: Error finding container 6d54dada1b1df7462c6b074cc47ef71dfc06691a229855ea425da49f9f842df5: Status 404 returned error can't find the container with id 6d54dada1b1df7462c6b074cc47ef71dfc06691a229855ea425da49f9f842df5 Jul 02 20:34:02.565983 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.565956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:34:02.566118 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.566083 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:34:02.566178 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.566133 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:10.566118823 +0000 UTC m=+68.435173537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:34:02.739705 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.739686 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tnqkm_7f096e16-c852-477d-bcac-de5ae629e0be/dns-node-resolver/0.log" Jul 02 20:34:02.767061 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.767032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:02.767197 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.767158 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:34:02.767197 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.767178 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f75b44cf7-kppff: secret "image-registry-tls" not found Jul 02 20:34:02.767310 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:02.767241 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls podName:8102a931-a235-47f9-82b2-7b5d1d5d1cbc nodeName:}" failed. No retries permitted until 2026-07-02 20:34:10.76722115 +0000 UTC m=+68.636275883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls") pod "image-registry-7f75b44cf7-kppff" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc") : secret "image-registry-tls" not found Jul 02 20:34:02.985279 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:02.985214 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7pb47" Jul 02 20:34:03.062074 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:03.062050 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" event={"ID":"9aafc3ae-4262-4207-974f-527fc8915031","Type":"ContainerStarted","Data":"6d54dada1b1df7462c6b074cc47ef71dfc06691a229855ea425da49f9f842df5"} Jul 02 20:34:03.063720 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:03.063684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" event={"ID":"b6f76bd4-923a-4861-86fe-f9846254277c","Type":"ContainerStarted","Data":"76d7ccef3dfbf4945e428689d2567c6bfe16378a4a0f24d9d77f706146465b0e"} Jul 02 20:34:03.080914 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:03.080870 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-qdsht" podStartSLOduration=1.344221103 podStartE2EDuration="4.080852217s" totalImagesPullingTime="2.736631114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:03.079869241 +0000 UTC m=+60.948923977" watchObservedRunningTime="2026-07-02 20:34:03.080852217 +0000 UTC m=+60.949906963" Jul 02 20:34:03.340701 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:03.340674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7f85s_a85dfcbb-c1a1-424a-82ec-83c502cd88b3/node-ca/0.log" Jul 02 20:34:04.139214 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:04.139195 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4k6j5_751d0909-edfc-4b48-9019-c83d5baea0cb/serve-healthcheck-canary/0.log" Jul 02 20:34:04.279074 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:04.279030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:04.279074 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:04.279070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:04.279191 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:04.279168 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:34:04.279225 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:04.279218 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:08.279204979 +0000 UTC m=+66.148259696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : secret "insights-runtime-extractor-tls" not found Jul 02 20:34:04.279265 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:04.279218 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:34:04.279296 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:04.279268 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:08.279252719 +0000 UTC m=+66.148307433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : configmap "kube-rbac-proxy" not found Jul 02 20:34:05.069965 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:05.069925 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" event={"ID":"9aafc3ae-4262-4207-974f-527fc8915031","Type":"ContainerStarted","Data":"fe4c1aee7357dd03c35d1ec2cc66254a017dab9b034a271e809bfd7ea289441f"} Jul 02 20:34:05.070347 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:05.069993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" event={"ID":"9aafc3ae-4262-4207-974f-527fc8915031","Type":"ContainerStarted","Data":"68e53c400ed6e4c8d32e3eaf1ce03b8d4851dbc60832635202a5545d7910b7f9"} Jul 02 20:34:05.090712 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:05.090656 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-mcm2m" podStartSLOduration=3.421383702 podStartE2EDuration="5.090640126s" totalImagesPullingTime="1.669256424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:05.08867717 +0000 UTC m=+62.957731906" watchObservedRunningTime="2026-07-02 20:34:05.090640126 +0000 UTC m=+62.959694924" Jul 02 20:34:08.309693 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.309657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:08.310190 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.309705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:08.310190 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.309815 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:34:08.310190 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.309884 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:16.309862079 +0000 UTC m=+74.178916805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : configmap "kube-rbac-proxy" not found Jul 02 20:34:08.310190 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.309898 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:34:08.310190 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.310042 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls podName:04a93c10-beba-4d5a-bcf3-e7a4fa222f92 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:16.309988712 +0000 UTC m=+74.179043442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g7jf" (UID: "04a93c10-beba-4d5a-bcf3-e7a4fa222f92") : secret "insights-runtime-extractor-tls" not found Jul 02 20:34:08.411891 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.411821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:34:08.412035 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.411978 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:34:08.415642 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.415617 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:34:08.415759 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.415637 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:34:08.422724 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.422704 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:34:08.422784 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:08.422756 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs podName:a7ca8736-448b-4aeb-bd02-d42cd0f778f4 nodeName:}" failed. No retries permitted until 2026-07-02 20:35:12.422742898 +0000 UTC m=+130.291797614 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs") pod "network-metrics-daemon-hn2fv" (UID: "a7ca8736-448b-4aeb-bd02-d42cd0f778f4") : secret "metrics-daemon-secret" not found Jul 02 20:34:08.425093 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.425064 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74f5fa0b-75e9-4866-afea-d08a6d24e683-original-pull-secret\") pod \"global-pull-secret-syncer-qdqwm\" (UID: \"74f5fa0b-75e9-4866-afea-d08a6d24e683\") " pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:34:08.512593 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.512569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:34:08.514816 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.514796 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxnr\" (UniqueName: \"kubernetes.io/projected/8aa26210-1768-4a93-91ba-ac9bf7ea6318-kube-api-access-vcxnr\") pod \"network-check-target-w5ssn\" (UID: \"8aa26210-1768-4a93-91ba-ac9bf7ea6318\") " pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:34:08.596129 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.596064 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdqwm" Jul 02 20:34:08.605836 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.605812 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w2d5j\"" Jul 02 20:34:08.613478 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.613431 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:34:08.731563 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.731536 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdqwm"] Jul 02 20:34:08.734338 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:08.734315 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f5fa0b_75e9_4866_afea_d08a6d24e683.slice/crio-89ea9658ee5fa41641364ffc130a004ea4d7e0ec179c7fe1155c1c3140d4f3a1 WatchSource:0}: Error finding container 89ea9658ee5fa41641364ffc130a004ea4d7e0ec179c7fe1155c1c3140d4f3a1: Status 404 returned error can't find the container with id 89ea9658ee5fa41641364ffc130a004ea4d7e0ec179c7fe1155c1c3140d4f3a1 Jul 02 20:34:08.750193 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:08.750174 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w5ssn"] Jul 02 20:34:08.752658 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:08.752637 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa26210_1768_4a93_91ba_ac9bf7ea6318.slice/crio-f6acb0e2253c6a86d369cbbcf2d9ea2dfadbe9146ba876baafb182eafac07d44 WatchSource:0}: Error finding container f6acb0e2253c6a86d369cbbcf2d9ea2dfadbe9146ba876baafb182eafac07d44: Status 404 returned error can't find the container with id f6acb0e2253c6a86d369cbbcf2d9ea2dfadbe9146ba876baafb182eafac07d44 Jul 02 20:34:09.082294 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:09.082254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5ssn" event={"ID":"8aa26210-1768-4a93-91ba-ac9bf7ea6318","Type":"ContainerStarted","Data":"89ea3092991d017e92ae7fbf79e6a86da81a0a254dca6fd9e6cf5ad2225993e5"} Jul 02 20:34:09.082426 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:09.082312 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w5ssn" event={"ID":"8aa26210-1768-4a93-91ba-ac9bf7ea6318","Type":"ContainerStarted","Data":"f6acb0e2253c6a86d369cbbcf2d9ea2dfadbe9146ba876baafb182eafac07d44"} Jul 02 20:34:09.082426 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:09.082367 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:34:09.083787 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:09.083762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdqwm" event={"ID":"74f5fa0b-75e9-4866-afea-d08a6d24e683","Type":"ContainerStarted","Data":"89ea9658ee5fa41641364ffc130a004ea4d7e0ec179c7fe1155c1c3140d4f3a1"} Jul 02 20:34:09.101105 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:09.101057 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w5ssn" podStartSLOduration=67.101042937 podStartE2EDuration="1m7.101042937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:09.100437004 +0000 UTC m=+66.969491739" watchObservedRunningTime="2026-07-02 20:34:09.101042937 +0000 UTC m=+66.970097705" Jul 02 20:34:10.329776 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.329740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:34:10.332393 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.332364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ac5aff-ebdd-415c-8b7d-46af46993d65-metrics-tls\") pod \"dns-default-lk8nz\" (UID: \"36ac5aff-ebdd-415c-8b7d-46af46993d65\") " pod="openshift-dns/dns-default-lk8nz" Jul 02 20:34:10.570335 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.570302 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w7dxd\"" Jul 02 20:34:10.578765 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.578720 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lk8nz" Jul 02 20:34:10.632993 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.632957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:34:10.633189 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:10.633142 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:34:10.633259 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:10.633224 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls podName:907b49ba-ac93-4909-8998-671f55aa7697 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:26.63319348 +0000 UTC m=+84.502248209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-6zc8f" (UID: "907b49ba-ac93-4909-8998-671f55aa7697") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:34:10.724615 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.724585 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lk8nz"] Jul 02 20:34:10.728180 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:10.728150 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ac5aff_ebdd_415c_8b7d_46af46993d65.slice/crio-92ebe5028ff2540cf595fa1cc12e104414f22024f01cf0016258b60d1f4757e4 WatchSource:0}: Error finding container 92ebe5028ff2540cf595fa1cc12e104414f22024f01cf0016258b60d1f4757e4: Status 404 returned error can't find the container with id 92ebe5028ff2540cf595fa1cc12e104414f22024f01cf0016258b60d1f4757e4 Jul 02 20:34:10.834901 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.834870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:10.845443 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.845390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"image-registry-7f75b44cf7-kppff\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:10.948257 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.948234 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kxfdk\"" Jul 02 20:34:10.956641 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:10.956623 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:11.091377 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:11.091341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk8nz" event={"ID":"36ac5aff-ebdd-415c-8b7d-46af46993d65","Type":"ContainerStarted","Data":"92ebe5028ff2540cf595fa1cc12e104414f22024f01cf0016258b60d1f4757e4"} Jul 02 20:34:12.483142 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:12.483116 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:34:12.486020 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:12.485995 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8102a931_a235_47f9_82b2_7b5d1d5d1cbc.slice/crio-ad6aeab846c48a6769fdc42570b95e3d010f164c19e1d81b799b71fb81b76350 WatchSource:0}: Error finding container ad6aeab846c48a6769fdc42570b95e3d010f164c19e1d81b799b71fb81b76350: Status 404 returned error can't find the container with id ad6aeab846c48a6769fdc42570b95e3d010f164c19e1d81b799b71fb81b76350 Jul 02 20:34:13.100178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.100136 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" event={"ID":"8102a931-a235-47f9-82b2-7b5d1d5d1cbc","Type":"ContainerStarted","Data":"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9"} Jul 02 20:34:13.100351 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.100206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" event={"ID":"8102a931-a235-47f9-82b2-7b5d1d5d1cbc","Type":"ContainerStarted","Data":"ad6aeab846c48a6769fdc42570b95e3d010f164c19e1d81b799b71fb81b76350"} Jul 02 20:34:13.100351 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.100233 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:13.102153 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.102129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdqwm" event={"ID":"74f5fa0b-75e9-4866-afea-d08a6d24e683","Type":"ContainerStarted","Data":"2d7feda47e80f91bd0800128f32f9df4e598ceba28f442ff20842d6503df9373"} Jul 02 20:34:13.120475 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.120395 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" podStartSLOduration=19.12037855 podStartE2EDuration="19.12037855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:13.120124061 +0000 UTC m=+70.989178795" watchObservedRunningTime="2026-07-02 20:34:13.12037855 +0000 UTC m=+70.989433285" Jul 02 20:34:13.135411 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:13.135358 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qdqwm" podStartSLOduration=67.14289749 podStartE2EDuration="1m11.135340556s" totalImagesPullingTime="3.992443066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:13.134367241 +0000 UTC m=+71.003421977" watchObservedRunningTime="2026-07-02 20:34:13.135340556 +0000 UTC m=+71.004395290" Jul 02 20:34:14.106766 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:14.106730 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk8nz" event={"ID":"36ac5aff-ebdd-415c-8b7d-46af46993d65","Type":"ContainerStarted","Data":"2713b56caa291106fc417fe8009a567cf4ecaee9fcd779ce00ab02689f557bbe"} Jul 02 20:34:14.107139 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:14.106791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lk8nz" event={"ID":"36ac5aff-ebdd-415c-8b7d-46af46993d65","Type":"ContainerStarted","Data":"c20b981f7dba5e39e0b5d5970fc8644644dcfdf052bfe12d36879f5684b065d5"} Jul 02 20:34:14.125601 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:14.125516 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lk8nz" podStartSLOduration=32.995982014 podStartE2EDuration="36.125500079s" totalImagesPullingTime="3.129518065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:14.125206155 +0000 UTC m=+71.994260883" watchObservedRunningTime="2026-07-02 20:34:14.125500079 +0000 UTC m=+71.994554818" Jul 02 20:34:15.110100 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:15.110071 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lk8nz" Jul 02 20:34:16.377056 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.377016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:16.377378 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.377086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:16.377560 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.377539 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:16.379407 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.379388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04a93c10-beba-4d5a-bcf3-e7a4fa222f92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g7jf\" (UID: \"04a93c10-beba-4d5a-bcf3-e7a4fa222f92\") " pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:16.476907 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.476881 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6z6g\"" Jul 02 20:34:16.484247 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.484225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9g7jf" Jul 02 20:34:16.607937 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:16.607903 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9g7jf"] Jul 02 20:34:16.611374 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:16.611347 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a93c10_beba_4d5a_bcf3_e7a4fa222f92.slice/crio-a6409db680f8424991e3a64b7fc1784184d7fea5cc3271e3020309889feb6eb0 WatchSource:0}: Error finding container a6409db680f8424991e3a64b7fc1784184d7fea5cc3271e3020309889feb6eb0: Status 404 returned error can't find the container with id a6409db680f8424991e3a64b7fc1784184d7fea5cc3271e3020309889feb6eb0 Jul 02 20:34:17.118316 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:17.118281 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g7jf" event={"ID":"04a93c10-beba-4d5a-bcf3-e7a4fa222f92","Type":"ContainerStarted","Data":"a6409db680f8424991e3a64b7fc1784184d7fea5cc3271e3020309889feb6eb0"} Jul 02 20:34:21.135867 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:21.135807 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g7jf" event={"ID":"04a93c10-beba-4d5a-bcf3-e7a4fa222f92","Type":"ContainerStarted","Data":"bd407128c14308e1d1f9c768bc80222a530c249921574a3235b540f37ccc5065"} Jul 02 20:34:21.136148 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:21.135870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g7jf" event={"ID":"04a93c10-beba-4d5a-bcf3-e7a4fa222f92","Type":"ContainerStarted","Data":"2d5518f018217794ddf063985068d5b527f2162a50f65293411f63ad66193fe6"} Jul 02 20:34:23.964785 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.964753 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n"] Jul 02 20:34:23.968136 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.968115 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:23.971334 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.971310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 02 20:34:23.971480 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.971367 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 20:34:23.971480 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.971424 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 02 20:34:23.971688 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.971667 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 20:34:23.972667 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.972649 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 20:34:23.972772 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.972673 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 02 20:34:23.972772 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.972673 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 02 20:34:23.985241 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.985213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n"] Jul 02 20:34:23.994841 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:23.994820 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:34:24.127510 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb44a74-7270-4244-8a9e-d9eea623e3be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.127634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.127634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127549 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.127634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.127634 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.127790 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.127655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/4eb44a74-7270-4244-8a9e-d9eea623e3be-kube-api-access-fqp2q\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.146536 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.146505 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g7jf" event={"ID":"04a93c10-beba-4d5a-bcf3-e7a4fa222f92","Type":"ContainerStarted","Data":"6ff9883d80acc199de6ce56fedaeb49d12604809bd4751d4886f9f997af7f219"} Jul 02 20:34:24.167174 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.167124 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9g7jf" podStartSLOduration=17.740846448 podStartE2EDuration="24.16711019s" totalImagesPullingTime="6.426263742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:34:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:24.166688457 +0000 UTC m=+82.035743191" watchObservedRunningTime="2026-07-02 20:34:24.16711019 +0000 UTC m=+82.036164924" Jul 02 20:34:24.228653 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb44a74-7270-4244-8a9e-d9eea623e3be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.228653 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.228779 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.228779 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.228779 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.228931 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.228864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/4eb44a74-7270-4244-8a9e-d9eea623e3be-kube-api-access-fqp2q\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.229482 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.229417 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb44a74-7270-4244-8a9e-d9eea623e3be-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.231331 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.231301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.231509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.231490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-ca\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.231870 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.231853 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.231935 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.231914 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb44a74-7270-4244-8a9e-d9eea623e3be-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.239412 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.239393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/4eb44a74-7270-4244-8a9e-d9eea623e3be-kube-api-access-fqp2q\") pod \"cluster-proxy-proxy-agent-57c9f5496-kdw2n\" (UID: \"4eb44a74-7270-4244-8a9e-d9eea623e3be\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.287863 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.287837 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" Jul 02 20:34:24.405258 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:24.405227 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n"] Jul 02 20:34:24.408065 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:24.408035 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb44a74_7270_4244_8a9e_d9eea623e3be.slice/crio-a45fcfc5bba45034c08eb01f139aa295c8b9fa77761da316f3a7bf7147445371 WatchSource:0}: Error finding container a45fcfc5bba45034c08eb01f139aa295c8b9fa77761da316f3a7bf7147445371: Status 404 returned error can't find the container with id a45fcfc5bba45034c08eb01f139aa295c8b9fa77761da316f3a7bf7147445371 Jul 02 20:34:25.116631 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:25.116597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lk8nz" Jul 02 20:34:25.153469 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:25.153418 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" event={"ID":"4eb44a74-7270-4244-8a9e-d9eea623e3be","Type":"ContainerStarted","Data":"a45fcfc5bba45034c08eb01f139aa295c8b9fa77761da316f3a7bf7147445371"} Jul 02 20:34:26.647753 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:26.647717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:34:26.650387 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:26.650361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/907b49ba-ac93-4909-8998-671f55aa7697-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-6zc8f\" (UID: \"907b49ba-ac93-4909-8998-671f55aa7697\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:34:26.921661 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:26.921582 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tlg6n\"" Jul 02 20:34:26.929088 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:26.929062 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" Jul 02 20:34:27.844673 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:27.844651 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f"] Jul 02 20:34:27.847228 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:27.847203 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907b49ba_ac93_4909_8998_671f55aa7697.slice/crio-d3daf9367630c51563d36084eb2c6c5232182195c651e548bf0ad335e4f311b9 WatchSource:0}: Error finding container d3daf9367630c51563d36084eb2c6c5232182195c651e548bf0ad335e4f311b9: Status 404 returned error can't find the container with id d3daf9367630c51563d36084eb2c6c5232182195c651e548bf0ad335e4f311b9 Jul 02 20:34:28.164606 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:28.164573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" event={"ID":"907b49ba-ac93-4909-8998-671f55aa7697","Type":"ContainerStarted","Data":"d3daf9367630c51563d36084eb2c6c5232182195c651e548bf0ad335e4f311b9"} Jul 02 20:34:28.166710 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:28.166684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" event={"ID":"4eb44a74-7270-4244-8a9e-d9eea623e3be","Type":"ContainerStarted","Data":"1beff983260ad061b9e499ea6a6fa979bfa8d8c8c40c022ad853d57e062caf82"} Jul 02 20:34:28.166810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:28.166720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" event={"ID":"4eb44a74-7270-4244-8a9e-d9eea623e3be","Type":"ContainerStarted","Data":"26c565f3b7f80c83cd21a8358e45c6904381528e1819b22e1879c5f17f2b4a69"} Jul 02 20:34:28.166810 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:28.166733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" event={"ID":"4eb44a74-7270-4244-8a9e-d9eea623e3be","Type":"ContainerStarted","Data":"9068378319ee473b46ae9c340425983c18a602c85410dec2d2ac8a84563719b1"} Jul 02 20:34:28.185798 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:28.185750 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c9f5496-kdw2n" podStartSLOduration=1.7943451289999999 podStartE2EDuration="5.185729063s" totalImagesPullingTime="3.391383934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:28.184976781 +0000 UTC m=+86.054031516" watchObservedRunningTime="2026-07-02 20:34:28.185729063 +0000 UTC m=+86.054783797" Jul 02 20:34:30.174734 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:30.174695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" event={"ID":"907b49ba-ac93-4909-8998-671f55aa7697","Type":"ContainerStarted","Data":"c94b11b7d60705a7ae296bb1885e6f835291a12ef4f5b91c503041e8b8864071"} Jul 02 20:34:30.194221 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:30.194177 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-6zc8f" podStartSLOduration=34.467549134 podStartE2EDuration="36.194163896s" totalImagesPullingTime="1.726614762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:30.192694486 +0000 UTC m=+88.061749221" watchObservedRunningTime="2026-07-02 20:34:30.194163896 +0000 UTC m=+88.063218608" Jul 02 20:34:33.999918 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:33.999892 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:38.520757 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.520723 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74wfr"] Jul 02 20:34:38.524166 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.524151 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.526897 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.526872 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 20:34:38.527029 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.527011 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 20:34:38.527875 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.527860 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l8f5g\"" Jul 02 20:34:38.528162 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.528147 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 20:34:38.529666 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.529647 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 20:34:38.625129 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-metrics-client-ca\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625243 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-sys\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625243 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcll5\" (UniqueName: \"kubernetes.io/projected/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-kube-api-access-wcll5\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625243 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-accelerators-collector-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625386 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625249 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-root\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625386 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-textfile\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625386 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625285 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625386 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.625386 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.625329 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-wtmp\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726053 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-metrics-client-ca\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-sys\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcll5\" (UniqueName: \"kubernetes.io/projected/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-kube-api-access-wcll5\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-accelerators-collector-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-root\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-sys\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726172 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-textfile\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-root\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-wtmp\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:38.726337 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:38.726402 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls podName:3f061dd2-5c34-486c-ac8c-9dd94cb51a01 nodeName:}" failed. No retries permitted until 2026-07-02 20:34:39.226385467 +0000 UTC m=+97.095440179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls") pod "node-exporter-74wfr" (UID: "3f061dd2-5c34-486c-ac8c-9dd94cb51a01") : secret "node-exporter-tls" not found Jul 02 20:34:38.726468 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726433 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-wtmp\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726796 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-textfile\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726796 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726742 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-metrics-client-ca\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.726796 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.726742 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-accelerators-collector-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.728504 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.728485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:38.748308 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:38.748279 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcll5\" (UniqueName: \"kubernetes.io/projected/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-kube-api-access-wcll5\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:39.230138 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:39.230114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:39.232333 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:39.232307 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f061dd2-5c34-486c-ac8c-9dd94cb51a01-node-exporter-tls\") pod \"node-exporter-74wfr\" (UID: \"3f061dd2-5c34-486c-ac8c-9dd94cb51a01\") " pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:39.432544 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:39.432522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74wfr" Jul 02 20:34:39.442251 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:34:39.442222 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f061dd2_5c34_486c_ac8c_9dd94cb51a01.slice/crio-56a8813f9b4b11433211b690f2ca091fb3f0968ed96e75b6ddb6acf26ebe693f WatchSource:0}: Error finding container 56a8813f9b4b11433211b690f2ca091fb3f0968ed96e75b6ddb6acf26ebe693f: Status 404 returned error can't find the container with id 56a8813f9b4b11433211b690f2ca091fb3f0968ed96e75b6ddb6acf26ebe693f Jul 02 20:34:40.089160 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:40.089122 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w5ssn" Jul 02 20:34:40.210028 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:40.209961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74wfr" event={"ID":"3f061dd2-5c34-486c-ac8c-9dd94cb51a01","Type":"ContainerStarted","Data":"56a8813f9b4b11433211b690f2ca091fb3f0968ed96e75b6ddb6acf26ebe693f"} Jul 02 20:34:41.214605 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:41.214578 2569 generic.go:356] "Generic (PLEG): container finished" podID="3f061dd2-5c34-486c-ac8c-9dd94cb51a01" containerID="e0c05b5f96248013e55f58ac610791aeff8781f77c936588c516e2626c2959b3" exitCode=0 Jul 02 20:34:41.214955 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:41.214656 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74wfr" event={"ID":"3f061dd2-5c34-486c-ac8c-9dd94cb51a01","Type":"ContainerDied","Data":"e0c05b5f96248013e55f58ac610791aeff8781f77c936588c516e2626c2959b3"} Jul 02 20:34:42.223091 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:42.223061 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74wfr" event={"ID":"3f061dd2-5c34-486c-ac8c-9dd94cb51a01","Type":"ContainerStarted","Data":"013e97d35b8ff083d09beef2da57acb30a18cc9bc456bab462f67154a3ab3cbb"} Jul 02 20:34:42.223516 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:42.223143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74wfr" event={"ID":"3f061dd2-5c34-486c-ac8c-9dd94cb51a01","Type":"ContainerStarted","Data":"ded1d1a77e230b78772f1c9beb3ec4fd8cd24b895fa12f41bc887690b1316245"} Jul 02 20:34:42.245062 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:42.245020 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74wfr" podStartSLOduration=3.304775412 podStartE2EDuration="4.245007717s" totalImagesPullingTime="878.371576ms" totalInitContainerRuntime="61.860729ms" isStatefulPod=true podCreationTimestamp="2026-07-02 20:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:42.244555823 +0000 UTC m=+100.113610556" watchObservedRunningTime="2026-07-02 20:34:42.245007717 +0000 UTC m=+100.114062450" Jul 02 20:34:49.013916 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.013857 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" podUID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" containerName="registry" containerID="cri-o://998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9" gracePeriod=30 Jul 02 20:34:49.245548 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.245518 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:49.246524 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.246497 2569 generic.go:356] "Generic (PLEG): container finished" podID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" containerID="998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9" exitCode=0 Jul 02 20:34:49.246620 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.246544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" event={"ID":"8102a931-a235-47f9-82b2-7b5d1d5d1cbc","Type":"ContainerDied","Data":"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9"} Jul 02 20:34:49.246620 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.246584 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" event={"ID":"8102a931-a235-47f9-82b2-7b5d1d5d1cbc","Type":"ContainerDied","Data":"ad6aeab846c48a6769fdc42570b95e3d010f164c19e1d81b799b71fb81b76350"} Jul 02 20:34:49.246620 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.246602 2569 scope.go:117] "RemoveContainer" containerID="998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9" Jul 02 20:34:49.254176 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.254161 2569 scope.go:117] "RemoveContainer" containerID="998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9" Jul 02 20:34:49.254466 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:34:49.254429 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9\": container with ID starting with 998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9 not found: ID does not exist" containerID="998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9" Jul 02 20:34:49.254516 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.254480 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9"} err="failed to get container status \"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9\": rpc error: code = NotFound desc = could not find container \"998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9\": container with ID starting with 998daf4ec8bf880e71592f87571dcfd45abd858b5d2c329141528afa256447c9 not found: ID does not exist" Jul 02 20:34:49.397542 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397509 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397555 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7grrb\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397617 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397690 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.397783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.397740 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted\") pod \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\" (UID: \"8102a931-a235-47f9-82b2-7b5d1d5d1cbc\") " Jul 02 20:34:49.398101 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.398038 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:34:49.398212 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.398154 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:34:49.400081 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.400053 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:34:49.400166 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.400083 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:34:49.400232 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.400211 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb" (OuterVolumeSpecName: "kube-api-access-7grrb") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "kube-api-access-7grrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:34:49.400274 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.400214 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:34:49.400274 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.400260 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:34:49.407252 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.407230 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8102a931-a235-47f9-82b2-7b5d1d5d1cbc" (UID: "8102a931-a235-47f9-82b2-7b5d1d5d1cbc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:34:49.498731 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498708 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-installation-pull-secrets\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498731 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498731 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-bound-sa-token\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498744 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498756 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-registry-certificates\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498767 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-ca-trust-extracted\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498781 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-image-registry-private-configuration\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498794 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-trusted-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:49.498860 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:49.498807 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grrb\" (UniqueName: \"kubernetes.io/projected/8102a931-a235-47f9-82b2-7b5d1d5d1cbc-kube-api-access-7grrb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:34:50.250786 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:50.250760 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f75b44cf7-kppff" Jul 02 20:34:50.274046 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:50.274020 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:34:50.276243 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:50.276220 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f75b44cf7-kppff"] Jul 02 20:34:50.779984 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:34:50.779965 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" path="/var/lib/kubelet/pods/8102a931-a235-47f9-82b2-7b5d1d5d1cbc/volumes" Jul 02 20:35:12.458244 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:12.458203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:35:12.460563 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:12.460539 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca8736-448b-4aeb-bd02-d42cd0f778f4-metrics-certs\") pod \"network-metrics-daemon-hn2fv\" (UID: \"a7ca8736-448b-4aeb-bd02-d42cd0f778f4\") " pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:35:12.489712 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:12.489679 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zddh5\"" Jul 02 20:35:12.496997 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:12.496976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn2fv" Jul 02 20:35:12.633643 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:12.633605 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn2fv"] Jul 02 20:35:12.636051 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:35:12.636025 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ca8736_448b_4aeb_bd02_d42cd0f778f4.slice/crio-85f5c62a0bb62347747f5dfb34db4526102c757efda0d2bbfe4e144bd7478eea WatchSource:0}: Error finding container 85f5c62a0bb62347747f5dfb34db4526102c757efda0d2bbfe4e144bd7478eea: Status 404 returned error can't find the container with id 85f5c62a0bb62347747f5dfb34db4526102c757efda0d2bbfe4e144bd7478eea Jul 02 20:35:13.324222 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:13.324185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2fv" event={"ID":"a7ca8736-448b-4aeb-bd02-d42cd0f778f4","Type":"ContainerStarted","Data":"85f5c62a0bb62347747f5dfb34db4526102c757efda0d2bbfe4e144bd7478eea"} Jul 02 20:35:14.329840 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:14.329802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2fv" event={"ID":"a7ca8736-448b-4aeb-bd02-d42cd0f778f4","Type":"ContainerStarted","Data":"fb9052416cabf397f186b648658733109263e08313728fcce136553650be45ba"} Jul 02 20:35:14.330184 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:14.329858 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn2fv" event={"ID":"a7ca8736-448b-4aeb-bd02-d42cd0f778f4","Type":"ContainerStarted","Data":"853430f5a78737f55449c07c6bb9f6a41adec14b38ab096777eb21df4d7c287a"} Jul 02 20:35:14.347439 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:35:14.347390 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hn2fv" podStartSLOduration=131.470740279 podStartE2EDuration="2m12.347376092s" totalImagesPullingTime="876.635813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:35:14.345371946 +0000 UTC m=+132.214426679" watchObservedRunningTime="2026-07-02 20:35:14.347376092 +0000 UTC m=+132.216430882" Jul 02 20:37:43.409252 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.409215 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7gsfq"] Jul 02 20:37:43.409742 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.409478 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" containerName="registry" Jul 02 20:37:43.409742 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.409491 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" containerName="registry" Jul 02 20:37:43.409742 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.409540 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8102a931-a235-47f9-82b2-7b5d1d5d1cbc" containerName="registry" Jul 02 20:37:43.412205 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.412185 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.414826 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.414797 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 20:37:43.414826 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.414808 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 20:37:43.414826 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.414814 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fpzrv\"" Jul 02 20:37:43.420418 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.420391 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7gsfq"] Jul 02 20:37:43.564801 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.564771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.564923 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.564810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xqp\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-kube-api-access-b4xqp\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.665678 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.665604 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.665678 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.665650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xqp\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-kube-api-access-b4xqp\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.679774 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.679750 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-bound-sa-token\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.680083 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.680058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xqp\" (UniqueName: \"kubernetes.io/projected/90fbc84e-b97d-4c52-89f3-b705737d9e0b-kube-api-access-b4xqp\") pod \"cert-manager-55b4d954d5-7gsfq\" (UID: \"90fbc84e-b97d-4c52-89f3-b705737d9e0b\") " pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.722827 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.722799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7gsfq" Jul 02 20:37:43.847044 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:43.847016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7gsfq"] Jul 02 20:37:43.849221 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:37:43.849194 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90fbc84e_b97d_4c52_89f3_b705737d9e0b.slice/crio-ec7c368126e9529e549a5ccc53bf293f4d473f9f80e4290ca5b9834e5fc52216 WatchSource:0}: Error finding container ec7c368126e9529e549a5ccc53bf293f4d473f9f80e4290ca5b9834e5fc52216: Status 404 returned error can't find the container with id ec7c368126e9529e549a5ccc53bf293f4d473f9f80e4290ca5b9834e5fc52216 Jul 02 20:37:44.835041 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:44.834998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7gsfq" event={"ID":"90fbc84e-b97d-4c52-89f3-b705737d9e0b","Type":"ContainerStarted","Data":"ec7c368126e9529e549a5ccc53bf293f4d473f9f80e4290ca5b9834e5fc52216"} Jul 02 20:37:46.453434 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.453385 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf"] Jul 02 20:37:46.456796 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.456779 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.460522 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.460499 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-sms2v\"" Jul 02 20:37:46.461620 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.461601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 20:37:46.461706 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.461601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:37:46.467332 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.467307 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf"] Jul 02 20:37:46.585821 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.585790 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.585979 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.585845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwmd\" (UniqueName: \"kubernetes.io/projected/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-kube-api-access-jjwmd\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.686872 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.686834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.687030 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.686895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwmd\" (UniqueName: \"kubernetes.io/projected/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-kube-api-access-jjwmd\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.687226 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.687204 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.695872 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.695847 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwmd\" (UniqueName: \"kubernetes.io/projected/7fa38dd2-98bb-46ec-9888-2ff983e46fb5-kube-api-access-jjwmd\") pod \"openshift-lws-operator-fd8ccff4c-pd5bf\" (UID: \"7fa38dd2-98bb-46ec-9888-2ff983e46fb5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.767725 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.767665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" Jul 02 20:37:46.923055 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:46.923033 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf"] Jul 02 20:37:46.925877 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:37:46.925855 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fa38dd2_98bb_46ec_9888_2ff983e46fb5.slice/crio-eb10ed6fffa80ee8bf1c6eba93d7dd7f13946fb378db57e5e527dbcac2b71656 WatchSource:0}: Error finding container eb10ed6fffa80ee8bf1c6eba93d7dd7f13946fb378db57e5e527dbcac2b71656: Status 404 returned error can't find the container with id eb10ed6fffa80ee8bf1c6eba93d7dd7f13946fb378db57e5e527dbcac2b71656 Jul 02 20:37:47.847959 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:47.847919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7gsfq" event={"ID":"90fbc84e-b97d-4c52-89f3-b705737d9e0b","Type":"ContainerStarted","Data":"7ac48c6427abc97257b88f93c9bbf4ae2b3d8758e5bd93cde27e00b1a331d268"} Jul 02 20:37:47.849510 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:47.849482 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" event={"ID":"7fa38dd2-98bb-46ec-9888-2ff983e46fb5","Type":"ContainerStarted","Data":"eb10ed6fffa80ee8bf1c6eba93d7dd7f13946fb378db57e5e527dbcac2b71656"} Jul 02 20:37:47.865682 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:37:47.865631 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7gsfq" podStartSLOduration=1.8429444400000001 podStartE2EDuration="4.865614715s" totalImagesPullingTime="3.022670275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:37:47.863970743 +0000 UTC m=+285.733025476" watchObservedRunningTime="2026-07-02 20:37:47.865614715 +0000 UTC m=+285.734669448" Jul 02 20:38:02.517181 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:02.517148 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 20:38:02.902308 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:02.902276 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" event={"ID":"7fa38dd2-98bb-46ec-9888-2ff983e46fb5","Type":"ContainerStarted","Data":"d47d2964089b29b6991d0e3090e2e0896ce6876008cd2926c10ea031fd6bb4f9"} Jul 02 20:38:02.917650 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:02.917603 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pd5bf" podStartSLOduration=1.820439862 podStartE2EDuration="16.917584578s" totalImagesPullingTime="15.097144716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:38:02.917151377 +0000 UTC m=+300.786206113" watchObservedRunningTime="2026-07-02 20:38:02.917584578 +0000 UTC m=+300.786639313" Jul 02 20:38:17.886218 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.886182 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9"] Jul 02 20:38:17.893470 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.893417 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:17.900323 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.899635 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-df8b5\"" Jul 02 20:38:17.900323 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.899936 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 20:38:17.900323 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.900163 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 20:38:17.900872 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.900372 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 20:38:17.909356 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.909336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9"] Jul 02 20:38:17.982229 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.982207 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v47m\" (UniqueName: \"kubernetes.io/projected/dd6edd46-475d-46f0-8f7a-a62bbfa23709-kube-api-access-7v47m\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:17.982346 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.982236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-metrics-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:17.982346 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.982254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd6edd46-475d-46f0-8f7a-a62bbfa23709-manager-config\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:17.982346 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:17.982310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.082924 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.082903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.083018 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.082941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v47m\" (UniqueName: \"kubernetes.io/projected/dd6edd46-475d-46f0-8f7a-a62bbfa23709-kube-api-access-7v47m\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.083018 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.082960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-metrics-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.083018 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.082982 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd6edd46-475d-46f0-8f7a-a62bbfa23709-manager-config\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.083631 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.083606 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dd6edd46-475d-46f0-8f7a-a62bbfa23709-manager-config\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.085237 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.085216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.085323 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.085262 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd6edd46-475d-46f0-8f7a-a62bbfa23709-metrics-cert\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.098284 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.098265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v47m\" (UniqueName: \"kubernetes.io/projected/dd6edd46-475d-46f0-8f7a-a62bbfa23709-kube-api-access-7v47m\") pod \"lws-controller-manager-6d6cb4dc5f-hmvp9\" (UID: \"dd6edd46-475d-46f0-8f7a-a62bbfa23709\") " pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.204019 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.203971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:18.320988 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.320957 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9"] Jul 02 20:38:18.323239 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:38:18.323217 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6edd46_475d_46f0_8f7a_a62bbfa23709.slice/crio-4acc66edc927b34957354f77de03b659cadce5c969847048fb1e10ea467b3fb8 WatchSource:0}: Error finding container 4acc66edc927b34957354f77de03b659cadce5c969847048fb1e10ea467b3fb8: Status 404 returned error can't find the container with id 4acc66edc927b34957354f77de03b659cadce5c969847048fb1e10ea467b3fb8 Jul 02 20:38:18.325023 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.325006 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:38:18.660680 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.660657 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh"] Jul 02 20:38:18.663609 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.663594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.666301 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.666283 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 20:38:18.666301 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.666284 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 20:38:18.666429 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.666307 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 20:38:18.666661 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.666645 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cc4jb\"" Jul 02 20:38:18.666931 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.666916 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 20:38:18.681982 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.681963 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh"] Jul 02 20:38:18.687790 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.687770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.687889 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.687797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-manifests\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.687889 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.687821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.687889 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.687848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-tmp\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.688030 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.687893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9f4\" (UniqueName: \"kubernetes.io/projected/cea7a512-eecc-4ccd-9db2-b02ab92c267f-kube-api-access-tr9f4\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789130 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789222 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-tmp\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789222 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9f4\" (UniqueName: \"kubernetes.io/projected/cea7a512-eecc-4ccd-9db2-b02ab92c267f-kube-api-access-tr9f4\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789298 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789298 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-manifests\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.789605 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.789585 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-manifests\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.791356 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.791335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea7a512-eecc-4ccd-9db2-b02ab92c267f-tmp\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.791528 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.791512 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-webhook-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.791591 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.791577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cea7a512-eecc-4ccd-9db2-b02ab92c267f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.802140 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.802119 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9f4\" (UniqueName: \"kubernetes.io/projected/cea7a512-eecc-4ccd-9db2-b02ab92c267f-kube-api-access-tr9f4\") pod \"opendatahub-operator-controller-manager-857d77d4-74hsh\" (UID: \"cea7a512-eecc-4ccd-9db2-b02ab92c267f\") " pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:18.952223 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.952169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" event={"ID":"dd6edd46-475d-46f0-8f7a-a62bbfa23709","Type":"ContainerStarted","Data":"4acc66edc927b34957354f77de03b659cadce5c969847048fb1e10ea467b3fb8"} Jul 02 20:38:18.973531 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:18.973508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:19.103907 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:19.103878 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh"] Jul 02 20:38:19.107345 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:38:19.107319 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea7a512_eecc_4ccd_9db2_b02ab92c267f.slice/crio-38ba73debccf124b90dc55c8bb75fc5dc203d361884e38c954861faf9b12a29c WatchSource:0}: Error finding container 38ba73debccf124b90dc55c8bb75fc5dc203d361884e38c954861faf9b12a29c: Status 404 returned error can't find the container with id 38ba73debccf124b90dc55c8bb75fc5dc203d361884e38c954861faf9b12a29c Jul 02 20:38:19.958778 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:19.958736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" event={"ID":"cea7a512-eecc-4ccd-9db2-b02ab92c267f","Type":"ContainerStarted","Data":"38ba73debccf124b90dc55c8bb75fc5dc203d361884e38c954861faf9b12a29c"} Jul 02 20:38:22.976925 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:22.976887 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" event={"ID":"dd6edd46-475d-46f0-8f7a-a62bbfa23709","Type":"ContainerStarted","Data":"63abbfa56e2e6176b40a86d8b44f18d76f9d8c0571607a437c7276597e8d7bbf"} Jul 02 20:38:22.977312 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:22.977022 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:22.978332 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:22.978306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" event={"ID":"cea7a512-eecc-4ccd-9db2-b02ab92c267f","Type":"ContainerStarted","Data":"1a9953b6e3814d8172e8cb1ee15eb816015c35836bedfd42d900ddab225d3c54"} Jul 02 20:38:22.994516 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:22.994474 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" podStartSLOduration=2.19659364 podStartE2EDuration="5.994444749s" totalImagesPullingTime="3.797851109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:38:22.993359086 +0000 UTC m=+320.862413822" watchObservedRunningTime="2026-07-02 20:38:22.994444749 +0000 UTC m=+320.863499482" Jul 02 20:38:23.257038 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:38:23.257014 2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea7a512_eecc_4ccd_9db2_b02ab92c267f.slice/crio-1a9953b6e3814d8172e8cb1ee15eb816015c35836bedfd42d900ddab225d3c54.scope\": RecentStats: unable to find data in memory cache]" Jul 02 20:38:23.983067 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:23.983035 2569 generic.go:356] "Generic (PLEG): container finished" podID="cea7a512-eecc-4ccd-9db2-b02ab92c267f" containerID="1a9953b6e3814d8172e8cb1ee15eb816015c35836bedfd42d900ddab225d3c54" exitCode=0 Jul 02 20:38:23.983479 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:23.983124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" event={"ID":"cea7a512-eecc-4ccd-9db2-b02ab92c267f","Type":"ContainerDied","Data":"1a9953b6e3814d8172e8cb1ee15eb816015c35836bedfd42d900ddab225d3c54"} Jul 02 20:38:24.988897 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:24.988856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" event={"ID":"cea7a512-eecc-4ccd-9db2-b02ab92c267f","Type":"ContainerStarted","Data":"257d647dccb319dd1ddc8ded139bd77e7000e25a6d78a5cd5c5c3717d7a465d6"} Jul 02 20:38:24.989377 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:24.988973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:25.013590 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:25.013537 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" podStartSLOduration=2.5069751829999998 podStartE2EDuration="7.013523274s" totalImagesPullingTime="3.463667876s" totalInitContainerRuntime="1.042880215s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:38:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:38:25.011338219 +0000 UTC m=+322.880392952" watchObservedRunningTime="2026-07-02 20:38:25.013523274 +0000 UTC m=+322.882578010" Jul 02 20:38:33.984954 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:33.984923 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d6cb4dc5f-hmvp9" Jul 02 20:38:35.994500 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:35.994445 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-857d77d4-74hsh" Jul 02 20:38:45.991462 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:45.991403 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr"] Jul 02 20:38:45.998647 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:45.998622 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.001617 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.001585 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 20:38:46.001734 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.001681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 20:38:46.001734 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.001691 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 20:38:46.002138 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.002118 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 20:38:46.002401 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.002383 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 20:38:46.002583 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.002561 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7w7vc\"" Jul 02 20:38:46.002981 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.002962 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 20:38:46.007915 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.007885 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr"] Jul 02 20:38:46.074986 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.074961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075105 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.074994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075105 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.075014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075105 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.075077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075252 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.075118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075252 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.075153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.075252 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.075177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgg8\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-kube-api-access-rrgg8\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175472 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175596 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175596 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175596 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175596 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175596 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.175848 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.175717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgg8\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-kube-api-access-rrgg8\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.176805 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.176780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.178334 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.178301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.182629 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.179054 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.182629 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.179249 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.182629 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.179770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.184889 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.184863 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgg8\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-kube-api-access-rrgg8\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.185422 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.185396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841e547a-c257-4e5e-9f5b-4cfaa9837ed8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vs8pr\" (UID: \"841e547a-c257-4e5e-9f5b-4cfaa9837ed8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.311012 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.310993 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:46.446047 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:46.446020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr"] Jul 02 20:38:46.448051 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:38:46.448026 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841e547a_c257_4e5e_9f5b_4cfaa9837ed8.slice/crio-33a69fee604aa18de77fb6fbb9eab413f2ffb816873ef77558caff495f382f4b WatchSource:0}: Error finding container 33a69fee604aa18de77fb6fbb9eab413f2ffb816873ef77558caff495f382f4b: Status 404 returned error can't find the container with id 33a69fee604aa18de77fb6fbb9eab413f2ffb816873ef77558caff495f382f4b Jul 02 20:38:47.070730 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:47.070697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" event={"ID":"841e547a-c257-4e5e-9f5b-4cfaa9837ed8","Type":"ContainerStarted","Data":"33a69fee604aa18de77fb6fbb9eab413f2ffb816873ef77558caff495f382f4b"} Jul 02 20:38:48.556658 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:48.556620 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 02 20:38:48.556923 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:48.556690 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 02 20:38:49.081568 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:49.081520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" event={"ID":"841e547a-c257-4e5e-9f5b-4cfaa9837ed8","Type":"ContainerStarted","Data":"f20d56991bb1f881565631accdb382b3081543ec3b0594f9975802b8cd5759af"} Jul 02 20:38:49.081790 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:49.081743 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:38:49.082811 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:49.082784 2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vs8pr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 20:38:49.082886 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:49.082844 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" podUID="841e547a-c257-4e5e-9f5b-4cfaa9837ed8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 20:38:49.153219 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:49.153170 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" podStartSLOduration=2.046633395 podStartE2EDuration="4.15315631s" totalImagesPullingTime="2.106522915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:38:49.14385676 +0000 UTC m=+347.012911495" watchObservedRunningTime="2026-07-02 20:38:49.15315631 +0000 UTC m=+347.022211046" Jul 02 20:38:50.086177 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:38:50.086151 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vs8pr" Jul 02 20:41:18.450759 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.450723 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:41:18.453783 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.453767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.456552 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.456529 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 20:41:18.458174 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.458155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzlr4\"" Jul 02 20:41:18.458264 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.458219 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 20:41:18.458314 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.458261 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 20:41:18.463823 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.463795 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:41:18.475919 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.475899 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:41:18.639972 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.639944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvv7b\" (UniqueName: \"kubernetes.io/projected/cb186070-d25c-453b-8dd7-5f65b13ef2ed-kube-api-access-wvv7b\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.640076 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.639979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb186070-d25c-453b-8dd7-5f65b13ef2ed-config-file\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.741183 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.741125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvv7b\" (UniqueName: \"kubernetes.io/projected/cb186070-d25c-453b-8dd7-5f65b13ef2ed-kube-api-access-wvv7b\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.741183 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.741158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb186070-d25c-453b-8dd7-5f65b13ef2ed-config-file\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.741778 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.741759 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cb186070-d25c-453b-8dd7-5f65b13ef2ed-config-file\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.749742 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.749722 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvv7b\" (UniqueName: \"kubernetes.io/projected/cb186070-d25c-453b-8dd7-5f65b13ef2ed-kube-api-access-wvv7b\") pod \"limitador-limitador-54d6d988d5-m2dqr\" (UID: \"cb186070-d25c-453b-8dd7-5f65b13ef2ed\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.763350 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.763326 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:18.886407 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:18.886383 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:41:18.889069 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:41:18.889041 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb186070_d25c_453b_8dd7_5f65b13ef2ed.slice/crio-9c5bfadeea566c65873122e875142aa7a075b800caf0b018574e323078b350a3 WatchSource:0}: Error finding container 9c5bfadeea566c65873122e875142aa7a075b800caf0b018574e323078b350a3: Status 404 returned error can't find the container with id 9c5bfadeea566c65873122e875142aa7a075b800caf0b018574e323078b350a3 Jul 02 20:41:19.671533 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:19.671497 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" event={"ID":"cb186070-d25c-453b-8dd7-5f65b13ef2ed","Type":"ContainerStarted","Data":"9c5bfadeea566c65873122e875142aa7a075b800caf0b018574e323078b350a3"} Jul 02 20:41:21.683280 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:21.683197 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" event={"ID":"cb186070-d25c-453b-8dd7-5f65b13ef2ed","Type":"ContainerStarted","Data":"720c5519b39263c33badb812c675566630f4388041ecbf9e4a942ea09d218f40"} Jul 02 20:41:21.683730 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:21.683288 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:41:21.700722 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:21.700676 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" podStartSLOduration=1.301607738 podStartE2EDuration="3.700659981s" totalImagesPullingTime="2.399052243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:41:21.699440554 +0000 UTC m=+499.568495289" watchObservedRunningTime="2026-07-02 20:41:21.700659981 +0000 UTC m=+499.569714716" Jul 02 20:41:32.688256 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:41:32.688221 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m2dqr" Jul 02 20:42:29.305188 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:42:29.305158 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:42:51.982608 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:42:51.982567 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:43:01.090076 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:01.090042 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:43:11.090583 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:11.090545 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:43:14.782553 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.782517 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d"] Jul 02 20:43:14.785705 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.785686 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.789998 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.789972 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gtvps\"" Jul 02 20:43:14.790363 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.790333 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 20:43:14.790499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.790403 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 02 20:43:14.790499 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.790461 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 20:43:14.793042 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.793017 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d"] Jul 02 20:43:14.911946 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.911910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74bac2b2-a23c-4803-8878-bb2e841f24d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.912106 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.911979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szsrt\" (UniqueName: \"kubernetes.io/projected/74bac2b2-a23c-4803-8878-bb2e841f24d3-kube-api-access-szsrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.912106 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.912043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.912106 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.912077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.912106 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.912102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:14.912304 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:14.912173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012512 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012638 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74bac2b2-a23c-4803-8878-bb2e841f24d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.012878 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsrt\" (UniqueName: \"kubernetes.io/projected/74bac2b2-a23c-4803-8878-bb2e841f24d3-kube-api-access-szsrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.013000 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.012974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.013145 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.013114 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.013301 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.013144 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.015214 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.015185 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74bac2b2-a23c-4803-8878-bb2e841f24d3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.015520 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.015496 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74bac2b2-a23c-4803-8878-bb2e841f24d3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.022309 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.022278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsrt\" (UniqueName: \"kubernetes.io/projected/74bac2b2-a23c-4803-8878-bb2e841f24d3-kube-api-access-szsrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d\" (UID: \"74bac2b2-a23c-4803-8878-bb2e841f24d3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.098675 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.098621 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:15.227443 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:15.227419 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d"] Jul 02 20:43:15.230182 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:43:15.230152 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bac2b2_a23c_4803_8878_bb2e841f24d3.slice/crio-e96d84bd25430d40adfaeb54545069c286d80d5fd0e93060b61042e0a2064b10 WatchSource:0}: Error finding container e96d84bd25430d40adfaeb54545069c286d80d5fd0e93060b61042e0a2064b10: Status 404 returned error can't find the container with id e96d84bd25430d40adfaeb54545069c286d80d5fd0e93060b61042e0a2064b10 Jul 02 20:43:16.139066 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:16.139027 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" event={"ID":"74bac2b2-a23c-4803-8878-bb2e841f24d3","Type":"ContainerStarted","Data":"e96d84bd25430d40adfaeb54545069c286d80d5fd0e93060b61042e0a2064b10"} Jul 02 20:43:16.581829 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:16.581799 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:43:17.143809 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:17.143778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" event={"ID":"74bac2b2-a23c-4803-8878-bb2e841f24d3","Type":"ContainerStarted","Data":"699d11716c5b61deff6f631593008ad6007bbb62ebc86c964637f5f4fa97c9da"} Jul 02 20:43:17.163509 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:17.163442 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" podStartSLOduration=2.17309018 podStartE2EDuration="3.163427846s" totalImagesPullingTime="990.337666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:43:17.161444747 +0000 UTC m=+615.030499483" watchObservedRunningTime="2026-07-02 20:43:17.163427846 +0000 UTC m=+615.032482618" Jul 02 20:43:25.099701 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:25.099594 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:25.099701 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:25.099668 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:25.112999 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:25.112973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:25.187420 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:25.187395 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d" Jul 02 20:43:34.996880 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:34.996830 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:43:46.489078 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:43:46.489037 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:45:29.495430 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:45:29.495390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:45:39.095439 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:45:39.095396 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:45:50.295758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:45:50.295723 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:45:59.894285 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:45:59.894242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:46:10.288357 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:46:10.288318 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:46:36.700716 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:46:36.700617 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:46:45.497732 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:46:45.497691 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:47:19.289943 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:47:19.289909 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:47:26.288703 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:47:26.288665 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:47:36.388129 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:47:36.388085 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:47:44.712436 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:47:44.712400 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:47:56.496643 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:47:56.496556 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:48:05.698065 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:48:05.698029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:48:17.286966 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:48:17.286930 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:48:26.390156 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:48:26.390118 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:48:49.799326 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:48:49.799287 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:48:58.890724 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:48:58.890686 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:49:23.489135 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:49:23.489099 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:49:33.690409 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:49:33.690325 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:49:40.088595 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:49:40.088561 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:49:49.886484 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:49:49.886431 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:50:00.487187 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:50:00.487148 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:50:09.502655 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:50:09.502614 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:50:20.096382 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:50:20.096348 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:50:29.389197 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:50:29.389161 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:52:17.230304 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:52:17.230254 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:52:28.196178 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:52:28.196081 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:52:36.988891 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:52:36.988849 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:52:48.197113 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:52:48.197072 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:52:57.796057 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:52:57.796019 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:53:09.386471 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:53:09.386411 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:53:18.197359 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:53:18.197321 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:53:28.983826 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:53:28.983790 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:53:40.293411 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:53:40.293373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:53:49.591344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:53:49.591295 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:54:00.187414 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:54:00.187320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:54:24.682260 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:54:24.682218 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:54:31.697813 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:54:31.697775 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:54:41.390163 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:54:41.390124 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:54:53.389734 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:54:53.389687 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:08.897475 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:08.897418 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:19.687702 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:19.687665 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:28.986770 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:28.986672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:38.591088 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:38.591050 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:47.491473 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:47.491423 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:55:58.383265 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:55:58.383222 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:56:12.297960 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:56:12.297921 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:56:21.196425 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:56:21.196389 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:56:30.596827 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:56:30.596777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:56:42.987344 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:56:42.987309 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:56:54.589824 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:56:54.589732 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:57:04.587892 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:57:04.587852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:57:15.892677 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:57:15.892638 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:57:27.294504 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:57:27.294446 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:57:37.892325 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:57:37.892283 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:57:51.597956 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:57:51.597915 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:02.584259 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:02.584230 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:11.789257 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:11.789214 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:26.060137 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.060105 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:26.063835 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.063812 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.066766 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.066745 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-eac2ac97-serving-cert\"" Jul 02 20:58:26.066924 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.066901 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 20:58:26.077972 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.077949 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:26.230618 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.230585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.230758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.230624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.230758 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.230652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ngw\" (UniqueName: \"kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.331391 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.331326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.331391 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.331359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.331391 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.331377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ngw\" (UniqueName: \"kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.332093 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.332071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.333916 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.333886 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.341733 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.341710 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ngw\" (UniqueName: \"kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw\") pod \"maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.374674 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.374651 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:26.500234 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.500203 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:26.502265 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:58:26.502240 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ab0a3a_e9c7_4074_b9e7_1dd92a23d91b.slice/crio-923564d8b6a8024acdecab6ff78e56bb11add91e91170c8381882d1f7b8d135c WatchSource:0}: Error finding container 923564d8b6a8024acdecab6ff78e56bb11add91e91170c8381882d1f7b8d135c: Status 404 returned error can't find the container with id 923564d8b6a8024acdecab6ff78e56bb11add91e91170c8381882d1f7b8d135c Jul 02 20:58:26.503626 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.503602 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:58:26.867318 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:26.867281 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" event={"ID":"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b","Type":"ContainerStarted","Data":"923564d8b6a8024acdecab6ff78e56bb11add91e91170c8381882d1f7b8d135c"} Jul 02 20:58:28.878909 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:28.878871 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" event={"ID":"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b","Type":"ContainerStarted","Data":"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b"} Jul 02 20:58:28.879288 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:28.879003 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:28.900693 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:28.900648 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" podStartSLOduration=1.589323325 podStartE2EDuration="2.900632586s" totalImagesPullingTime="1.311309261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:58:28.898875767 +0000 UTC m=+1526.767930502" watchObservedRunningTime="2026-07-02 20:58:28.900632586 +0000 UTC m=+1526.769687320" Jul 02 20:58:30.689887 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:30.689854 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:31.562213 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.562170 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:31.562423 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.562392 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" podUID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" containerName="maas-api" containerID="cri-o://eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b" gracePeriod=30 Jul 02 20:58:31.804011 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.803986 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:31.892614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.892545 2569 generic.go:356] "Generic (PLEG): container finished" podID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" containerID="eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b" exitCode=0 Jul 02 20:58:31.892614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.892588 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" Jul 02 20:58:31.892614 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.892595 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" event={"ID":"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b","Type":"ContainerDied","Data":"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b"} Jul 02 20:58:31.892792 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.892646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8" event={"ID":"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b","Type":"ContainerDied","Data":"923564d8b6a8024acdecab6ff78e56bb11add91e91170c8381882d1f7b8d135c"} Jul 02 20:58:31.892792 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.892676 2569 scope.go:117] "RemoveContainer" containerID="eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b" Jul 02 20:58:31.900231 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.900213 2569 scope.go:117] "RemoveContainer" containerID="eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b" Jul 02 20:58:31.900522 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:58:31.900503 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b\": container with ID starting with eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b not found: ID does not exist" containerID="eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b" Jul 02 20:58:31.900580 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.900531 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b"} err="failed to get container status \"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b\": rpc error: code = NotFound desc = could not find container \"eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b\": container with ID starting with eac5e6b9c22dea67b6539db31295f80c020727cf77ea127f48daeebc7ae91c0b not found: ID does not exist" Jul 02 20:58:31.973958 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.973937 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca\") pod \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " Jul 02 20:58:31.974059 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.973968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls\") pod \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " Jul 02 20:58:31.974059 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.973987 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5ngw\" (UniqueName: \"kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw\") pod \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\" (UID: \"30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b\") " Jul 02 20:58:31.974308 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.974279 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" (UID: "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:58:31.976039 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.976019 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" (UID: "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:58:31.976142 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:31.976115 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw" (OuterVolumeSpecName: "kube-api-access-w5ngw") pod "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" (UID: "30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b"). InnerVolumeSpecName "kube-api-access-w5ngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:32.075020 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.074997 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:32.075020 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.075018 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:32.075139 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.075027 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ngw\" (UniqueName: \"kubernetes.io/projected/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b-kube-api-access-w5ngw\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:32.213935 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.213909 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:32.217880 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.217859 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-eac2ac97-7c6759dbf8-vhcf8"] Jul 02 20:58:32.782397 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:32.782363 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" path="/var/lib/kubelet/pods/30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b/volumes" Jul 02 20:58:39.533683 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:39.533637 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:44.725424 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.725385 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:44.725814 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.725709 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" containerName="maas-api" Jul 02 20:58:44.725814 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.725722 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" containerName="maas-api" Jul 02 20:58:44.725814 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.725777 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ab0a3a-e9c7-4074-b9e7-1dd92a23d91b" containerName="maas-api" Jul 02 20:58:44.728862 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.728841 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.731665 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.731639 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-a3fd23bb-serving-cert\"" Jul 02 20:58:44.732352 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.732333 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 20:58:44.737289 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.737252 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:44.871776 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.871749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wh9\" (UniqueName: \"kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.871912 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.871788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.871912 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.871811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.973070 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.973046 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.973227 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.973120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wh9\" (UniqueName: \"kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.973300 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.973276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.973871 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.973850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.975508 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.975439 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:44.981747 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:44.981728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wh9\" (UniqueName: \"kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9\") pod \"maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:45.041428 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.041406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:45.165795 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.165768 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:45.168167 ip-10-0-140-111 kubenswrapper[2569]: W0702 20:58:45.168138 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3679efbd_4fed_41cd_b667_f088b278517f.slice/crio-b756ed0b3b3af95a1694443f6ad66a4a998926dd97149abf589e0ef5363d13af WatchSource:0}: Error finding container b756ed0b3b3af95a1694443f6ad66a4a998926dd97149abf589e0ef5363d13af: Status 404 returned error can't find the container with id b756ed0b3b3af95a1694443f6ad66a4a998926dd97149abf589e0ef5363d13af Jul 02 20:58:45.949307 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.949237 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" event={"ID":"3679efbd-4fed-41cd-b667-f088b278517f","Type":"ContainerStarted","Data":"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871"} Jul 02 20:58:45.949307 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.949282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" event={"ID":"3679efbd-4fed-41cd-b667-f088b278517f","Type":"ContainerStarted","Data":"b756ed0b3b3af95a1694443f6ad66a4a998926dd97149abf589e0ef5363d13af"} Jul 02 20:58:45.949729 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.949372 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:45.968706 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:45.968666 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" podStartSLOduration=1.57692517 podStartE2EDuration="1.968652443s" totalImagesPullingTime="391.727273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:58:45.967310687 +0000 UTC m=+1543.836365422" watchObservedRunningTime="2026-07-02 20:58:45.968652443 +0000 UTC m=+1543.837707177" Jul 02 20:58:48.083522 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:48.083483 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 20:58:49.482930 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.482879 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:49.483442 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.483147 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" podUID="3679efbd-4fed-41cd-b667-f088b278517f" containerName="maas-api" containerID="cri-o://d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871" gracePeriod=30 Jul 02 20:58:49.722555 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.722533 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:49.808400 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.808373 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca\") pod \"3679efbd-4fed-41cd-b667-f088b278517f\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " Jul 02 20:58:49.808550 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.808419 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls\") pod \"3679efbd-4fed-41cd-b667-f088b278517f\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " Jul 02 20:58:49.808550 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.808470 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2wh9\" (UniqueName: \"kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9\") pod \"3679efbd-4fed-41cd-b667-f088b278517f\" (UID: \"3679efbd-4fed-41cd-b667-f088b278517f\") " Jul 02 20:58:49.808801 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.808774 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3679efbd-4fed-41cd-b667-f088b278517f" (UID: "3679efbd-4fed-41cd-b667-f088b278517f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:58:49.810539 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.810516 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3679efbd-4fed-41cd-b667-f088b278517f" (UID: "3679efbd-4fed-41cd-b667-f088b278517f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:58:49.810629 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.810605 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9" (OuterVolumeSpecName: "kube-api-access-k2wh9") pod "3679efbd-4fed-41cd-b667-f088b278517f" (UID: "3679efbd-4fed-41cd-b667-f088b278517f"). InnerVolumeSpecName "kube-api-access-k2wh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:49.909628 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.909603 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3679efbd-4fed-41cd-b667-f088b278517f-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:49.909628 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.909627 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3679efbd-4fed-41cd-b667-f088b278517f-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:49.909753 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.909643 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wh9\" (UniqueName: \"kubernetes.io/projected/3679efbd-4fed-41cd-b667-f088b278517f-kube-api-access-k2wh9\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 20:58:49.966442 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.966416 2569 generic.go:356] "Generic (PLEG): container finished" podID="3679efbd-4fed-41cd-b667-f088b278517f" containerID="d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871" exitCode=0 Jul 02 20:58:49.966555 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.966472 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" Jul 02 20:58:49.966555 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.966480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" event={"ID":"3679efbd-4fed-41cd-b667-f088b278517f","Type":"ContainerDied","Data":"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871"} Jul 02 20:58:49.966555 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.966532 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px" event={"ID":"3679efbd-4fed-41cd-b667-f088b278517f","Type":"ContainerDied","Data":"b756ed0b3b3af95a1694443f6ad66a4a998926dd97149abf589e0ef5363d13af"} Jul 02 20:58:49.966706 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.966562 2569 scope.go:117] "RemoveContainer" containerID="d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871" Jul 02 20:58:49.974330 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.974289 2569 scope.go:117] "RemoveContainer" containerID="d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871" Jul 02 20:58:49.974604 ip-10-0-140-111 kubenswrapper[2569]: E0702 20:58:49.974584 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871\": container with ID starting with d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871 not found: ID does not exist" containerID="d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871" Jul 02 20:58:49.974674 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.974613 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871"} err="failed to get container status \"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871\": rpc error: code = NotFound desc = could not find container \"d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871\": container with ID starting with d77f655f416f6c6b81bd8c1f5c34f3641143f35e1c9ddb2997ba63f62edb8871 not found: ID does not exist" Jul 02 20:58:49.986979 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.986958 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:49.989005 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:49.988986 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-a3fd23bb-d4f5b5b85-c44px"] Jul 02 20:58:50.781896 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:50.781858 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3679efbd-4fed-41cd-b667-f088b278517f" path="/var/lib/kubelet/pods/3679efbd-4fed-41cd-b667-f088b278517f/volumes" Jul 02 20:58:57.083048 ip-10-0-140-111 kubenswrapper[2569]: I0702 20:58:57.083004 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:00:17.397975 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:17.397871 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:00:24.693817 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:24.693778 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:00:32.696289 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:32.691737 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:00:33.987702 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.987657 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:33.988106 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.988086 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3679efbd-4fed-41cd-b667-f088b278517f" containerName="maas-api" Jul 02 21:00:33.988155 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.988109 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3679efbd-4fed-41cd-b667-f088b278517f" containerName="maas-api" Jul 02 21:00:33.988207 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.988193 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3679efbd-4fed-41cd-b667-f088b278517f" containerName="maas-api" Jul 02 21:00:33.991599 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.991578 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:33.995729 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.995707 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 21:00:33.997565 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:33.997545 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-3ef915bc-serving-cert\"" Jul 02 21:00:34.010043 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.010017 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:34.039277 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.039253 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.039369 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.039299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.039369 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.039324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw4m\" (UniqueName: \"kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.140249 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.140220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.140377 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.140268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.140377 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.140292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw4m\" (UniqueName: \"kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.140473 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:00:34.140386 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-3ef915bc-serving-cert: secret "maas-api-e2e-mt-3ef915bc-serving-cert" not found Jul 02 21:00:34.140528 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:00:34.140513 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls podName:e53a56d1-89ca-4ddf-951b-ba5767f8d1a5 nodeName:}" failed. No retries permitted until 2026-07-02 21:00:34.640442361 +0000 UTC m=+1652.509497087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls") pod "maas-api-e2e-mt-3ef915bc-9496c5556-975hb" (UID: "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5") : secret "maas-api-e2e-mt-3ef915bc-serving-cert" not found Jul 02 21:00:34.140942 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.140923 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.153333 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.153310 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw4m\" (UniqueName: \"kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.643289 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.643258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.645805 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.645780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") pod \"maas-api-e2e-mt-3ef915bc-9496c5556-975hb\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:34.902613 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:34.902538 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:35.028014 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:35.027944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:35.030182 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:00:35.030153 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53a56d1_89ca_4ddf_951b_ba5767f8d1a5.slice/crio-1f95d51cbed340a5a7d143f2fc0e107f3c687b093b7187f546210c90cd45a57e WatchSource:0}: Error finding container 1f95d51cbed340a5a7d143f2fc0e107f3c687b093b7187f546210c90cd45a57e: Status 404 returned error can't find the container with id 1f95d51cbed340a5a7d143f2fc0e107f3c687b093b7187f546210c90cd45a57e Jul 02 21:00:35.412723 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:35.412690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" event={"ID":"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5","Type":"ContainerStarted","Data":"1f95d51cbed340a5a7d143f2fc0e107f3c687b093b7187f546210c90cd45a57e"} Jul 02 21:00:36.418643 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:36.418607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" event={"ID":"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5","Type":"ContainerStarted","Data":"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8"} Jul 02 21:00:36.419068 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:36.418673 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:36.435168 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:36.435117 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" podStartSLOduration=2.784362361 podStartE2EDuration="3.435097591s" totalImagesPullingTime="650.73523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:00:36.434719438 +0000 UTC m=+1654.303774173" watchObservedRunningTime="2026-07-02 21:00:36.435097591 +0000 UTC m=+1654.304152326" Jul 02 21:00:42.428929 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:42.428891 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:42.588178 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:42.588142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:00:46.146766 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.146727 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:46.147218 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.146954 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" podUID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" containerName="maas-api" containerID="cri-o://9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8" gracePeriod=30 Jul 02 21:00:46.387434 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.387408 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:46.432226 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.432175 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca\") pod \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " Jul 02 21:00:46.432314 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.432234 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvw4m\" (UniqueName: \"kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m\") pod \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " Jul 02 21:00:46.432314 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.432260 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") pod \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\" (UID: \"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5\") " Jul 02 21:00:46.432578 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.432551 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" (UID: "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:00:46.434134 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.434108 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m" (OuterVolumeSpecName: "kube-api-access-jvw4m") pod "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" (UID: "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5"). InnerVolumeSpecName "kube-api-access-jvw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:00:46.434240 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.434205 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" (UID: "e53a56d1-89ca-4ddf-951b-ba5767f8d1a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:00:46.464650 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.464621 2569 generic.go:356] "Generic (PLEG): container finished" podID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" containerID="9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8" exitCode=0 Jul 02 21:00:46.464748 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.464658 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" Jul 02 21:00:46.464748 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.464669 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" event={"ID":"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5","Type":"ContainerDied","Data":"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8"} Jul 02 21:00:46.464860 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.464747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb" event={"ID":"e53a56d1-89ca-4ddf-951b-ba5767f8d1a5","Type":"ContainerDied","Data":"1f95d51cbed340a5a7d143f2fc0e107f3c687b093b7187f546210c90cd45a57e"} Jul 02 21:00:46.464860 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.464780 2569 scope.go:117] "RemoveContainer" containerID="9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8" Jul 02 21:00:46.472955 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.472934 2569 scope.go:117] "RemoveContainer" containerID="9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8" Jul 02 21:00:46.473187 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:00:46.473168 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8\": container with ID starting with 9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8 not found: ID does not exist" containerID="9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8" Jul 02 21:00:46.473230 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.473193 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8"} err="failed to get container status \"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8\": rpc error: code = NotFound desc = could not find container \"9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8\": container with ID starting with 9e2f911db4436c923e3924d1be8e975703f753c01c6fbc991b0a9e504b7de2d8 not found: ID does not exist" Jul 02 21:00:46.493421 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.493398 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:46.506110 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.506090 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-3ef915bc-9496c5556-975hb"] Jul 02 21:00:46.533732 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.533713 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:00:46.533827 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.533734 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw4m\" (UniqueName: \"kubernetes.io/projected/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-kube-api-access-jvw4m\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:00:46.533827 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.533745 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:00:46.781576 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:46.781548 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" path="/var/lib/kubelet/pods/e53a56d1-89ca-4ddf-951b-ba5767f8d1a5/volumes" Jul 02 21:00:49.791905 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:00:49.791874 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:01:14.303998 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.303961 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:14.304632 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.304239 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" containerName="maas-api" Jul 02 21:01:14.304632 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.304251 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" containerName="maas-api" Jul 02 21:01:14.304632 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.304338 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53a56d1-89ca-4ddf-951b-ba5767f8d1a5" containerName="maas-api" Jul 02 21:01:14.307543 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.307515 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.310280 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.310245 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-e59ec025-serving-cert\"" Jul 02 21:01:14.310565 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.310547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 21:01:14.316337 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.316314 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:14.423533 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.423502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.423674 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.423550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.423674 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.423583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtm5j\" (UniqueName: \"kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.524196 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.524173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.524347 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.524210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.524347 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.524242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm5j\" (UniqueName: \"kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.524491 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:01:14.524350 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-e59ec025-serving-cert: secret "maas-api-e2e-mt-e59ec025-serving-cert" not found Jul 02 21:01:14.524491 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:01:14.524437 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls podName:b2a415db-76be-473a-9e28-958ff385b9b8 nodeName:}" failed. No retries permitted until 2026-07-02 21:01:15.024414614 +0000 UTC m=+1692.893469327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls") pod "maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" (UID: "b2a415db-76be-473a-9e28-958ff385b9b8") : secret "maas-api-e2e-mt-e59ec025-serving-cert" not found Jul 02 21:01:14.524816 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.524793 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:14.537362 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:14.537335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm5j\" (UniqueName: \"kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:15.028503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:15.028466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:15.030931 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:15.030898 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") pod \"maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:15.220633 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:15.220607 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:15.345881 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:15.345845 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:15.347932 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:01:15.347900 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a415db_76be_473a_9e28_958ff385b9b8.slice/crio-0e008ccaeb4a2ab28bced49b0b6b91e5c418fc86b0cceb893b725840061791c8 WatchSource:0}: Error finding container 0e008ccaeb4a2ab28bced49b0b6b91e5c418fc86b0cceb893b725840061791c8: Status 404 returned error can't find the container with id 0e008ccaeb4a2ab28bced49b0b6b91e5c418fc86b0cceb893b725840061791c8 Jul 02 21:01:15.591810 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:15.591748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" event={"ID":"b2a415db-76be-473a-9e28-958ff385b9b8","Type":"ContainerStarted","Data":"0e008ccaeb4a2ab28bced49b0b6b91e5c418fc86b0cceb893b725840061791c8"} Jul 02 21:01:16.598426 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:16.598392 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" event={"ID":"b2a415db-76be-473a-9e28-958ff385b9b8","Type":"ContainerStarted","Data":"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57"} Jul 02 21:01:16.598808 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:16.598526 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:16.619564 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:16.619522 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" podStartSLOduration=1.867469635 podStartE2EDuration="2.61950652s" totalImagesPullingTime="752.036885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:01:16.617409216 +0000 UTC m=+1694.486463953" watchObservedRunningTime="2026-07-02 21:01:16.61950652 +0000 UTC m=+1694.488561254" Jul 02 21:01:16.791325 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:16.791298 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:01:22.614253 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:22.614216 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:29.490937 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:29.490842 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:01:45.586079 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:45.586036 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:01:51.811555 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:51.811512 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:51.812009 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:51.811748 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" podUID="b2a415db-76be-473a-9e28-958ff385b9b8" containerName="maas-api" containerID="cri-o://608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57" gracePeriod=30 Jul 02 21:01:52.049629 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.049607 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:52.177211 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.177153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca\") pod \"b2a415db-76be-473a-9e28-958ff385b9b8\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " Jul 02 21:01:52.177211 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.177187 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtm5j\" (UniqueName: \"kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j\") pod \"b2a415db-76be-473a-9e28-958ff385b9b8\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " Jul 02 21:01:52.177347 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.177229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") pod \"b2a415db-76be-473a-9e28-958ff385b9b8\" (UID: \"b2a415db-76be-473a-9e28-958ff385b9b8\") " Jul 02 21:01:52.177611 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.177564 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2a415db-76be-473a-9e28-958ff385b9b8" (UID: "b2a415db-76be-473a-9e28-958ff385b9b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:01:52.179160 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.179139 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j" (OuterVolumeSpecName: "kube-api-access-mtm5j") pod "b2a415db-76be-473a-9e28-958ff385b9b8" (UID: "b2a415db-76be-473a-9e28-958ff385b9b8"). InnerVolumeSpecName "kube-api-access-mtm5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:01:52.179235 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.179162 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2a415db-76be-473a-9e28-958ff385b9b8" (UID: "b2a415db-76be-473a-9e28-958ff385b9b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:01:52.278247 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.278221 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a415db-76be-473a-9e28-958ff385b9b8-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:01:52.278247 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.278245 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtm5j\" (UniqueName: \"kubernetes.io/projected/b2a415db-76be-473a-9e28-958ff385b9b8-kube-api-access-mtm5j\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:01:52.278393 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.278254 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a415db-76be-473a-9e28-958ff385b9b8-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:01:52.764339 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.764266 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2a415db-76be-473a-9e28-958ff385b9b8" containerID="608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57" exitCode=0 Jul 02 21:01:52.764678 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.764582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" event={"ID":"b2a415db-76be-473a-9e28-958ff385b9b8","Type":"ContainerDied","Data":"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57"} Jul 02 21:01:52.764868 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.764791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" event={"ID":"b2a415db-76be-473a-9e28-958ff385b9b8","Type":"ContainerDied","Data":"0e008ccaeb4a2ab28bced49b0b6b91e5c418fc86b0cceb893b725840061791c8"} Jul 02 21:01:52.764868 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.764840 2569 scope.go:117] "RemoveContainer" containerID="608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57" Jul 02 21:01:52.765241 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.765198 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr" Jul 02 21:01:52.776136 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.776115 2569 scope.go:117] "RemoveContainer" containerID="608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57" Jul 02 21:01:52.776436 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:01:52.776411 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57\": container with ID starting with 608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57 not found: ID does not exist" containerID="608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57" Jul 02 21:01:52.776521 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.776470 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57"} err="failed to get container status \"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57\": rpc error: code = NotFound desc = could not find container \"608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57\": container with ID starting with 608e17c7ce5ed228281695b8a80a0aa161d96203e9d166dbb6efa6f5d4b8ee57 not found: ID does not exist" Jul 02 21:01:52.787520 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.787495 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:52.791906 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:52.791883 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-e59ec025-64dc99994b-wlgdr"] Jul 02 21:01:54.781469 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:54.781422 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a415db-76be-473a-9e28-958ff385b9b8" path="/var/lib/kubelet/pods/b2a415db-76be-473a-9e28-958ff385b9b8/volumes" Jul 02 21:01:55.095545 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:01:55.095470 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:02:05.989478 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:02:05.989415 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:02:42.284557 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:02:42.284522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:03:11.885175 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:03:11.885132 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:03:25.293194 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:03:25.293152 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:04:06.079892 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.079860 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:04:06.080391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.080163 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a415db-76be-473a-9e28-958ff385b9b8" containerName="maas-api" Jul 02 21:04:06.080391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.080177 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a415db-76be-473a-9e28-958ff385b9b8" containerName="maas-api" Jul 02 21:04:06.080391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.080231 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a415db-76be-473a-9e28-958ff385b9b8" containerName="maas-api" Jul 02 21:04:06.083229 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.083208 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.085814 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.085794 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-d9e8a3-serving-cert\"" Jul 02 21:04:06.085953 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.085914 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 21:04:06.092938 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.092911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:04:06.212695 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.212660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqg26\" (UniqueName: \"kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.212828 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.212705 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.212828 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.212739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.313184 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.313153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.313310 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.313208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqg26\" (UniqueName: \"kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.313310 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.313256 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.313310 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:04:06.313296 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-d9e8a3-serving-cert: secret "maas-api-e2e-inf-a-d9e8a3-serving-cert" not found Jul 02 21:04:06.313496 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:04:06.313369 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls podName:a3946281-9e8d-48d2-a335-6551a6a21a92 nodeName:}" failed. No retries permitted until 2026-07-02 21:04:06.813350742 +0000 UTC m=+1864.682405455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls") pod "maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" (UID: "a3946281-9e8d-48d2-a335-6551a6a21a92") : secret "maas-api-e2e-inf-a-d9e8a3-serving-cert" not found Jul 02 21:04:06.313953 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.313929 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.329553 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.329531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqg26\" (UniqueName: \"kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.817151 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.817117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.819687 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.819665 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:06.993923 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:06.993891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:07.121668 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:07.121525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:04:07.124309 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:04:07.124283 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3946281_9e8d_48d2_a335_6551a6a21a92.slice/crio-5dd4e4df8aca868af1efa617d2da456e7ef2ced52c90db77bd3dfc94d1510e84 WatchSource:0}: Error finding container 5dd4e4df8aca868af1efa617d2da456e7ef2ced52c90db77bd3dfc94d1510e84: Status 404 returned error can't find the container with id 5dd4e4df8aca868af1efa617d2da456e7ef2ced52c90db77bd3dfc94d1510e84 Jul 02 21:04:07.125933 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:07.125912 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 21:04:07.332537 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:07.332481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" event={"ID":"a3946281-9e8d-48d2-a335-6551a6a21a92","Type":"ContainerStarted","Data":"5dd4e4df8aca868af1efa617d2da456e7ef2ced52c90db77bd3dfc94d1510e84"} Jul 02 21:04:08.339049 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:08.339008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" event={"ID":"a3946281-9e8d-48d2-a335-6551a6a21a92","Type":"ContainerStarted","Data":"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d"} Jul 02 21:04:08.339495 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:08.339139 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:08.354936 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:08.354886 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" podStartSLOduration=1.8028708230000001 podStartE2EDuration="2.354870469s" totalImagesPullingTime="551.999646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:04:08.353969946 +0000 UTC m=+1866.223024681" watchObservedRunningTime="2026-07-02 21:04:08.354870469 +0000 UTC m=+1866.223925197" Jul 02 21:04:12.186833 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:12.186797 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:04:14.305610 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.305578 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:04:14.309322 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.309296 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.311978 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.311952 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-d34e2e-serving-cert\"" Jul 02 21:04:14.320562 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.320539 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:04:14.348784 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.348760 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:04:14.470072 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.470046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.470200 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.470109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7pfv\" (UniqueName: \"kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.470200 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.470134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.571090 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.571025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7pfv\" (UniqueName: \"kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.571090 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.571056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.571262 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.571094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.571262 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:04:14.571174 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-d34e2e-serving-cert: secret "maas-api-e2e-inf-b-d34e2e-serving-cert" not found Jul 02 21:04:14.571262 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:04:14.571243 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls podName:a07a0065-4bf3-480b-b0b5-62abd0df5c38 nodeName:}" failed. No retries permitted until 2026-07-02 21:04:15.071225412 +0000 UTC m=+1872.940280146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls") pod "maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" (UID: "a07a0065-4bf3-480b-b0b5-62abd0df5c38") : secret "maas-api-e2e-inf-b-d34e2e-serving-cert" not found Jul 02 21:04:14.571670 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.571653 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:14.579391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:14.579371 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7pfv\" (UniqueName: \"kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:15.075009 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:15.074980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:15.077553 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:15.077532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:15.220401 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:15.220376 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:15.343245 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:15.343213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:04:15.345824 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:04:15.345798 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07a0065_4bf3_480b_b0b5_62abd0df5c38.slice/crio-b5108791103db126e2d078336966edab8fd272a3779d7292f5633c2068633af2 WatchSource:0}: Error finding container b5108791103db126e2d078336966edab8fd272a3779d7292f5633c2068633af2: Status 404 returned error can't find the container with id b5108791103db126e2d078336966edab8fd272a3779d7292f5633c2068633af2 Jul 02 21:04:15.367072 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:15.367043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" event={"ID":"a07a0065-4bf3-480b-b0b5-62abd0df5c38","Type":"ContainerStarted","Data":"b5108791103db126e2d078336966edab8fd272a3779d7292f5633c2068633af2"} Jul 02 21:04:16.372897 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:16.372860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" event={"ID":"a07a0065-4bf3-480b-b0b5-62abd0df5c38","Type":"ContainerStarted","Data":"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401"} Jul 02 21:04:16.373267 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:16.372961 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:16.389740 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:16.389695 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" podStartSLOduration=1.9212147480000001 podStartE2EDuration="2.389678851s" totalImagesPullingTime="468.464103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:04:16.38832826 +0000 UTC m=+1874.257382994" watchObservedRunningTime="2026-07-02 21:04:16.389678851 +0000 UTC m=+1874.258733586" Jul 02 21:04:22.383264 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:22.383235 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:04:24.190166 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:24.190118 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:04:32.984671 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:32.984642 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:04:52.593547 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:04:52.593507 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:05:05.893819 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:05.893772 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:05:09.086536 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.086501 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:05:09.087007 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.086706 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerName="maas-api" containerID="cri-o://9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d" gracePeriod=30 Jul 02 21:05:09.344709 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.344621 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.26:8443/health\": dial tcp 10.134.0.26:8443: connect: connection refused" Jul 02 21:05:09.527778 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.527753 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:05:09.564645 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.564622 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca\") pod \"a3946281-9e8d-48d2-a335-6551a6a21a92\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " Jul 02 21:05:09.564742 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.564678 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") pod \"a3946281-9e8d-48d2-a335-6551a6a21a92\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " Jul 02 21:05:09.564742 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.564718 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqg26\" (UniqueName: \"kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26\") pod \"a3946281-9e8d-48d2-a335-6551a6a21a92\" (UID: \"a3946281-9e8d-48d2-a335-6551a6a21a92\") " Jul 02 21:05:09.564928 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.564906 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3946281-9e8d-48d2-a335-6551a6a21a92" (UID: "a3946281-9e8d-48d2-a335-6551a6a21a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:05:09.566650 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.566625 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26" (OuterVolumeSpecName: "kube-api-access-fqg26") pod "a3946281-9e8d-48d2-a335-6551a6a21a92" (UID: "a3946281-9e8d-48d2-a335-6551a6a21a92"). InnerVolumeSpecName "kube-api-access-fqg26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:05:09.566741 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.566714 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3946281-9e8d-48d2-a335-6551a6a21a92" (UID: "a3946281-9e8d-48d2-a335-6551a6a21a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:05:09.606762 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.606710 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerID="9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d" exitCode=0 Jul 02 21:05:09.606762 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.606749 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" Jul 02 21:05:09.606869 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.606764 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" event={"ID":"a3946281-9e8d-48d2-a335-6551a6a21a92","Type":"ContainerDied","Data":"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d"} Jul 02 21:05:09.606869 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.606803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d" event={"ID":"a3946281-9e8d-48d2-a335-6551a6a21a92","Type":"ContainerDied","Data":"5dd4e4df8aca868af1efa617d2da456e7ef2ced52c90db77bd3dfc94d1510e84"} Jul 02 21:05:09.606869 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.606824 2569 scope.go:117] "RemoveContainer" containerID="9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d" Jul 02 21:05:09.615278 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.615262 2569 scope.go:117] "RemoveContainer" containerID="9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d" Jul 02 21:05:09.615526 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:05:09.615507 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d\": container with ID starting with 9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d not found: ID does not exist" containerID="9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d" Jul 02 21:05:09.615578 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.615533 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d"} err="failed to get container status \"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d\": rpc error: code = NotFound desc = could not find container \"9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d\": container with ID starting with 9a84b607cf7cc561d19fd5faa3f9a33ab83c1f578e3c8f0301afb9e1ffe5dc2d not found: ID does not exist" Jul 02 21:05:09.627945 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.627921 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:05:09.634783 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.634762 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d9e8a3-d6b4bbd9d-lz55d"] Jul 02 21:05:09.666056 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.666037 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqg26\" (UniqueName: \"kubernetes.io/projected/a3946281-9e8d-48d2-a335-6551a6a21a92-kube-api-access-fqg26\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:09.666056 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.666056 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3946281-9e8d-48d2-a335-6551a6a21a92-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:09.666158 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:09.666067 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3946281-9e8d-48d2-a335-6551a6a21a92-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:10.782670 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:10.782636 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" path="/var/lib/kubelet/pods/a3946281-9e8d-48d2-a335-6551a6a21a92/volumes" Jul 02 21:05:19.789038 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:19.788998 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:05:51.754638 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:51.754608 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:05:51.755207 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:51.754823 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" podUID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" containerName="maas-api" containerID="cri-o://032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401" gracePeriod=30 Jul 02 21:05:52.191316 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.191293 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:05:52.263147 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.263121 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7pfv\" (UniqueName: \"kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv\") pod \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " Jul 02 21:05:52.263282 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.263162 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca\") pod \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " Jul 02 21:05:52.263282 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.263195 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") pod \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\" (UID: \"a07a0065-4bf3-480b-b0b5-62abd0df5c38\") " Jul 02 21:05:52.263622 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.263592 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a07a0065-4bf3-480b-b0b5-62abd0df5c38" (UID: "a07a0065-4bf3-480b-b0b5-62abd0df5c38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:05:52.265149 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.265129 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv" (OuterVolumeSpecName: "kube-api-access-q7pfv") pod "a07a0065-4bf3-480b-b0b5-62abd0df5c38" (UID: "a07a0065-4bf3-480b-b0b5-62abd0df5c38"). InnerVolumeSpecName "kube-api-access-q7pfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:05:52.265255 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.265163 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a07a0065-4bf3-480b-b0b5-62abd0df5c38" (UID: "a07a0065-4bf3-480b-b0b5-62abd0df5c38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:05:52.364649 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.364600 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7pfv\" (UniqueName: \"kubernetes.io/projected/a07a0065-4bf3-480b-b0b5-62abd0df5c38-kube-api-access-q7pfv\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:52.364649 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.364622 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07a0065-4bf3-480b-b0b5-62abd0df5c38-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:52.364649 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.364631 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a07a0065-4bf3-480b-b0b5-62abd0df5c38-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:05:52.788696 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.788665 2569 generic.go:356] "Generic (PLEG): container finished" podID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" containerID="032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401" exitCode=0 Jul 02 21:05:52.789131 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.788726 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" Jul 02 21:05:52.789131 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.788736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" event={"ID":"a07a0065-4bf3-480b-b0b5-62abd0df5c38","Type":"ContainerDied","Data":"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401"} Jul 02 21:05:52.789131 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.788782 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj" event={"ID":"a07a0065-4bf3-480b-b0b5-62abd0df5c38","Type":"ContainerDied","Data":"b5108791103db126e2d078336966edab8fd272a3779d7292f5633c2068633af2"} Jul 02 21:05:52.789131 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.788817 2569 scope.go:117] "RemoveContainer" containerID="032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401" Jul 02 21:05:52.797715 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.797696 2569 scope.go:117] "RemoveContainer" containerID="032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401" Jul 02 21:05:52.797999 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:05:52.797976 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401\": container with ID starting with 032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401 not found: ID does not exist" containerID="032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401" Jul 02 21:05:52.798037 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.798008 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401"} err="failed to get container status \"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401\": rpc error: code = NotFound desc = could not find container \"032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401\": container with ID starting with 032ec06d62cfce4495cdb83cecd3589008b3e3799f7fc97e7f28ba85a35b9401 not found: ID does not exist" Jul 02 21:05:52.805155 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.805128 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:05:52.809275 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:52.809250 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-d34e2e-55bbbfd6d8-5vtmj"] Jul 02 21:05:54.689511 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:54.689396 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:05:54.782376 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:05:54.782335 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" path="/var/lib/kubelet/pods/a07a0065-4bf3-480b-b0b5-62abd0df5c38/volumes" Jul 02 21:06:01.806218 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806184 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806621 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerName="maas-api" Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806640 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerName="maas-api" Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806662 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" containerName="maas-api" Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806671 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" containerName="maas-api" Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806748 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3946281-9e8d-48d2-a335-6551a6a21a92" containerName="maas-api" Jul 02 21:06:01.806801 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.806759 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a07a0065-4bf3-480b-b0b5-62abd0df5c38" containerName="maas-api" Jul 02 21:06:01.811027 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.811004 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:01.814051 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.814028 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-586d03-serving-cert\"" Jul 02 21:06:01.814188 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.814075 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 21:06:01.819521 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.819497 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:01.931739 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.931716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:01.931839 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.931743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtp44\" (UniqueName: \"kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:01.931839 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:01.931812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.032729 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.032705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.032838 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.032795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.032880 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.032834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtp44\" (UniqueName: \"kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.033415 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.033393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.035183 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.035161 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.043780 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.043762 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtp44\" (UniqueName: \"kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44\") pod \"maas-api-e2e-api-a-586d03-55f4659565-vjzbz\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.123065 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.123020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.246181 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.246153 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:02.248288 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:06:02.248260 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195a24be_2a51_4ad8_8c55_1998722c2619.slice/crio-1d0225784fb6958702708fd815ec60802263764ec3a1feef2cba591a85a5a73c WatchSource:0}: Error finding container 1d0225784fb6958702708fd815ec60802263764ec3a1feef2cba591a85a5a73c: Status 404 returned error can't find the container with id 1d0225784fb6958702708fd815ec60802263764ec3a1feef2cba591a85a5a73c Jul 02 21:06:02.836573 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.836538 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" event={"ID":"195a24be-2a51-4ad8-8c55-1998722c2619","Type":"ContainerStarted","Data":"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386"} Jul 02 21:06:02.836930 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.836595 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" event={"ID":"195a24be-2a51-4ad8-8c55-1998722c2619","Type":"ContainerStarted","Data":"1d0225784fb6958702708fd815ec60802263764ec3a1feef2cba591a85a5a73c"} Jul 02 21:06:02.836930 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.836643 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:02.872728 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:02.872651 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" podStartSLOduration=1.511312721 podStartE2EDuration="1.872636982s" totalImagesPullingTime="361.324261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:06:02.870549227 +0000 UTC m=+1980.739603962" watchObservedRunningTime="2026-07-02 21:06:02.872636982 +0000 UTC m=+1980.741691695" Jul 02 21:06:07.698234 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:07.698186 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:06:08.846363 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:08.846330 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:09.805148 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.805118 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:09.808285 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.808264 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.810832 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.810812 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-ff2295-serving-cert\"" Jul 02 21:06:09.818871 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.818851 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:09.893043 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.893017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxdx\" (UniqueName: \"kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.893346 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.893056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.893346 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.893085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.994088 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.994067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxdx\" (UniqueName: \"kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.994196 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.994107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.994238 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:06:09.994199 2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-ff2295-serving-cert: secret "maas-api-e2e-api-b-ff2295-serving-cert" not found Jul 02 21:06:09.994272 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.994235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:09.994272 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:06:09.994245 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls podName:0e17e82e-b264-4eae-83d0-9dcff4c39f6a nodeName:}" failed. No retries permitted until 2026-07-02 21:06:10.494230008 +0000 UTC m=+1988.363284721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls") pod "maas-api-e2e-api-b-ff2295-745f969996-rjcwl" (UID: "0e17e82e-b264-4eae-83d0-9dcff4c39f6a") : secret "maas-api-e2e-api-b-ff2295-serving-cert" not found Jul 02 21:06:09.994825 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:09.994808 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:10.004300 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.004279 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxdx\" (UniqueName: \"kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:10.499310 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.499279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:10.501848 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.501820 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") pod \"maas-api-e2e-api-b-ff2295-745f969996-rjcwl\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:10.719903 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.719875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:10.844187 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.844155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:10.846615 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:06:10.846587 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e17e82e_b264_4eae_83d0_9dcff4c39f6a.slice/crio-a4c42b73098bb1eb2239b3ed3f1736a67aa57247ffa643105473f61295ece925 WatchSource:0}: Error finding container a4c42b73098bb1eb2239b3ed3f1736a67aa57247ffa643105473f61295ece925: Status 404 returned error can't find the container with id a4c42b73098bb1eb2239b3ed3f1736a67aa57247ffa643105473f61295ece925 Jul 02 21:06:10.870815 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:10.870790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" event={"ID":"0e17e82e-b264-4eae-83d0-9dcff4c39f6a","Type":"ContainerStarted","Data":"a4c42b73098bb1eb2239b3ed3f1736a67aa57247ffa643105473f61295ece925"} Jul 02 21:06:11.877920 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:11.877880 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" event={"ID":"0e17e82e-b264-4eae-83d0-9dcff4c39f6a","Type":"ContainerStarted","Data":"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74"} Jul 02 21:06:11.878406 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:11.878014 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:11.894882 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:11.894830 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" podStartSLOduration=2.6015525630000003 podStartE2EDuration="2.894811816s" totalImagesPullingTime="293.259253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:06:11.893929263 +0000 UTC m=+1989.762983999" watchObservedRunningTime="2026-07-02 21:06:11.894811816 +0000 UTC m=+1989.763866551" Jul 02 21:06:17.886899 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:17.886863 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:19.987147 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:19.987117 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:06:23.207790 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.207760 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:23.208236 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.207991 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" podUID="195a24be-2a51-4ad8-8c55-1998722c2619" containerName="maas-api" containerID="cri-o://3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386" gracePeriod=30 Jul 02 21:06:23.449418 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.449394 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:23.588806 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.588769 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtp44\" (UniqueName: \"kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44\") pod \"195a24be-2a51-4ad8-8c55-1998722c2619\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " Jul 02 21:06:23.588955 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.588822 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca\") pod \"195a24be-2a51-4ad8-8c55-1998722c2619\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " Jul 02 21:06:23.588955 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.588887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls\") pod \"195a24be-2a51-4ad8-8c55-1998722c2619\" (UID: \"195a24be-2a51-4ad8-8c55-1998722c2619\") " Jul 02 21:06:23.589146 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.589119 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "195a24be-2a51-4ad8-8c55-1998722c2619" (UID: "195a24be-2a51-4ad8-8c55-1998722c2619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:06:23.590839 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.590808 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "195a24be-2a51-4ad8-8c55-1998722c2619" (UID: "195a24be-2a51-4ad8-8c55-1998722c2619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:06:23.590956 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.590919 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44" (OuterVolumeSpecName: "kube-api-access-rtp44") pod "195a24be-2a51-4ad8-8c55-1998722c2619" (UID: "195a24be-2a51-4ad8-8c55-1998722c2619"). InnerVolumeSpecName "kube-api-access-rtp44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:06:23.689301 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.689273 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195a24be-2a51-4ad8-8c55-1998722c2619-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:23.689301 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.689304 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtp44\" (UniqueName: \"kubernetes.io/projected/195a24be-2a51-4ad8-8c55-1998722c2619-kube-api-access-rtp44\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:23.689470 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.689319 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195a24be-2a51-4ad8-8c55-1998722c2619-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:23.928636 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.928570 2569 generic.go:356] "Generic (PLEG): container finished" podID="195a24be-2a51-4ad8-8c55-1998722c2619" containerID="3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386" exitCode=0 Jul 02 21:06:23.928636 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.928627 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" Jul 02 21:06:23.928765 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.928664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" event={"ID":"195a24be-2a51-4ad8-8c55-1998722c2619","Type":"ContainerDied","Data":"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386"} Jul 02 21:06:23.928765 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.928727 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz" event={"ID":"195a24be-2a51-4ad8-8c55-1998722c2619","Type":"ContainerDied","Data":"1d0225784fb6958702708fd815ec60802263764ec3a1feef2cba591a85a5a73c"} Jul 02 21:06:23.928765 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.928760 2569 scope.go:117] "RemoveContainer" containerID="3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386" Jul 02 21:06:23.938332 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.938312 2569 scope.go:117] "RemoveContainer" containerID="3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386" Jul 02 21:06:23.938668 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:06:23.938644 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386\": container with ID starting with 3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386 not found: ID does not exist" containerID="3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386" Jul 02 21:06:23.938721 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.938677 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386"} err="failed to get container status \"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386\": rpc error: code = NotFound desc = could not find container \"3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386\": container with ID starting with 3967baac353e2447dde18818aa7a48a29915046b88bd3d072348ae8870ba9386 not found: ID does not exist" Jul 02 21:06:23.949818 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.949791 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:23.953069 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:23.953044 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-586d03-55f4659565-vjzbz"] Jul 02 21:06:24.781124 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:24.781082 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195a24be-2a51-4ad8-8c55-1998722c2619" path="/var/lib/kubelet/pods/195a24be-2a51-4ad8-8c55-1998722c2619/volumes" Jul 02 21:06:30.027387 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.027357 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:30.027851 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.027626 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" podUID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" containerName="maas-api" containerID="cri-o://d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74" gracePeriod=30 Jul 02 21:06:30.278653 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.278590 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:30.440642 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.440611 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca\") pod \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " Jul 02 21:06:30.440804 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.440655 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") pod \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " Jul 02 21:06:30.440804 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.440715 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dxdx\" (UniqueName: \"kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx\") pod \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\" (UID: \"0e17e82e-b264-4eae-83d0-9dcff4c39f6a\") " Jul 02 21:06:30.440968 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.440943 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e17e82e-b264-4eae-83d0-9dcff4c39f6a" (UID: "0e17e82e-b264-4eae-83d0-9dcff4c39f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:06:30.442803 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.442778 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e17e82e-b264-4eae-83d0-9dcff4c39f6a" (UID: "0e17e82e-b264-4eae-83d0-9dcff4c39f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:06:30.442803 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.442785 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx" (OuterVolumeSpecName: "kube-api-access-7dxdx") pod "0e17e82e-b264-4eae-83d0-9dcff4c39f6a" (UID: "0e17e82e-b264-4eae-83d0-9dcff4c39f6a"). InnerVolumeSpecName "kube-api-access-7dxdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:06:30.541797 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.541766 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:30.541797 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.541795 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:30.541989 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.541806 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dxdx\" (UniqueName: \"kubernetes.io/projected/0e17e82e-b264-4eae-83d0-9dcff4c39f6a-kube-api-access-7dxdx\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:06:30.965039 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.964953 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" containerID="d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74" exitCode=0 Jul 02 21:06:30.965039 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.964998 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" Jul 02 21:06:30.965208 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.965037 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" event={"ID":"0e17e82e-b264-4eae-83d0-9dcff4c39f6a","Type":"ContainerDied","Data":"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74"} Jul 02 21:06:30.965208 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.965090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl" event={"ID":"0e17e82e-b264-4eae-83d0-9dcff4c39f6a","Type":"ContainerDied","Data":"a4c42b73098bb1eb2239b3ed3f1736a67aa57247ffa643105473f61295ece925"} Jul 02 21:06:30.965208 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.965115 2569 scope.go:117] "RemoveContainer" containerID="d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74" Jul 02 21:06:30.973749 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.973731 2569 scope.go:117] "RemoveContainer" containerID="d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74" Jul 02 21:06:30.974009 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:06:30.973988 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74\": container with ID starting with d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74 not found: ID does not exist" containerID="d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74" Jul 02 21:06:30.974076 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.974018 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74"} err="failed to get container status \"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74\": rpc error: code = NotFound desc = could not find container \"d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74\": container with ID starting with d3d9cc664f41767f929198e27b22a741154f27c82df5bacb8e80c27f935c1e74 not found: ID does not exist" Jul 02 21:06:30.980846 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.980823 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:30.984736 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:30.984715 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-ff2295-745f969996-rjcwl"] Jul 02 21:06:32.781276 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:32.781242 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" path="/var/lib/kubelet/pods/0e17e82e-b264-4eae-83d0-9dcff4c39f6a/volumes" Jul 02 21:06:33.193604 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:33.193518 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:06:40.045979 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.045941 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:06:40.046417 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046338 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" containerName="maas-api" Jul 02 21:06:40.046417 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046356 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" containerName="maas-api" Jul 02 21:06:40.046417 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046372 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195a24be-2a51-4ad8-8c55-1998722c2619" containerName="maas-api" Jul 02 21:06:40.046417 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046381 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="195a24be-2a51-4ad8-8c55-1998722c2619" containerName="maas-api" Jul 02 21:06:40.046647 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046513 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="195a24be-2a51-4ad8-8c55-1998722c2619" containerName="maas-api" Jul 02 21:06:40.046647 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.046531 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e17e82e-b264-4eae-83d0-9dcff4c39f6a" containerName="maas-api" Jul 02 21:06:40.049184 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.049163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.052609 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.052586 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-ffa3c8-serving-cert\"" Jul 02 21:06:40.052835 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.052819 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-k4crx\"" Jul 02 21:06:40.062989 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.062965 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:06:40.215680 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.215651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.215831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.215693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r894d\" (UniqueName: \"kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.215831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.215721 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.316967 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.316906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.317086 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.316986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.317086 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.317015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r894d\" (UniqueName: \"kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.317406 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.317388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.319291 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.319272 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.325633 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.325607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r894d\" (UniqueName: \"kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d\") pod \"maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.358532 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.358505 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:40.490084 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:40.489949 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:06:40.492703 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:06:40.492665 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71a6ecc_359c_487f_95ab_6e898e54e8ed.slice/crio-f121312578e618954dda8381cebfacf539e7688ffdad99716bc3c893c292f3a2 WatchSource:0}: Error finding container f121312578e618954dda8381cebfacf539e7688ffdad99716bc3c893c292f3a2: Status 404 returned error can't find the container with id f121312578e618954dda8381cebfacf539e7688ffdad99716bc3c893c292f3a2 Jul 02 21:06:41.007113 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:41.007070 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" event={"ID":"f71a6ecc-359c-487f-95ab-6e898e54e8ed","Type":"ContainerStarted","Data":"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e"} Jul 02 21:06:41.007270 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:41.007149 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" event={"ID":"f71a6ecc-359c-487f-95ab-6e898e54e8ed","Type":"ContainerStarted","Data":"f121312578e618954dda8381cebfacf539e7688ffdad99716bc3c893c292f3a2"} Jul 02 21:06:41.007270 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:41.007186 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:41.023547 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:41.023504 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" podStartSLOduration=0.654434923 podStartE2EDuration="1.023490636s" totalImagesPullingTime="369.055713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:06:41.022054433 +0000 UTC m=+2018.891109211" watchObservedRunningTime="2026-07-02 21:06:41.023490636 +0000 UTC m=+2018.892545371" Jul 02 21:06:44.492193 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:44.492155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:06:47.016230 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:47.016195 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:06:48.098985 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.098954 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:06:48.102314 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.102288 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.105877 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.105856 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-b780eb-serving-cert\"" Jul 02 21:06:48.122649 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.122626 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:06:48.278476 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.278417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.278653 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.278557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbxdn\" (UniqueName: \"kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.278653 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.278609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.379976 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.379887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxdn\" (UniqueName: \"kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.379976 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.379930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.379976 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.379949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.380578 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.380559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.382255 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.382236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.388634 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.388607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxdn\" (UniqueName: \"kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn\") pod \"maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.414790 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.414761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:48.542296 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:48.542266 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:06:48.545471 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:06:48.545422 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac95532_d903_4e29_8b9a_f0d9df74d843.slice/crio-a227db7d4769f9493e7c7e61696ed760750786f56b55fd90ce51ddf8cf631f48 WatchSource:0}: Error finding container a227db7d4769f9493e7c7e61696ed760750786f56b55fd90ce51ddf8cf631f48: Status 404 returned error can't find the container with id a227db7d4769f9493e7c7e61696ed760750786f56b55fd90ce51ddf8cf631f48 Jul 02 21:06:49.047938 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:49.047905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" event={"ID":"cac95532-d903-4e29-8b9a-f0d9df74d843","Type":"ContainerStarted","Data":"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f"} Jul 02 21:06:49.048104 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:49.047958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" event={"ID":"cac95532-d903-4e29-8b9a-f0d9df74d843","Type":"ContainerStarted","Data":"a227db7d4769f9493e7c7e61696ed760750786f56b55fd90ce51ddf8cf631f48"} Jul 02 21:06:49.048104 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:49.048067 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:06:49.066223 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:49.066175 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" podStartSLOduration=0.733978149 podStartE2EDuration="1.066159564s" totalImagesPullingTime="332.181415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:06:49.064527651 +0000 UTC m=+2026.933582386" watchObservedRunningTime="2026-07-02 21:06:49.066159564 +0000 UTC m=+2026.935214299" Jul 02 21:06:51.699077 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:51.699041 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:06:55.057034 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:06:55.057007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:07:06.307345 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:07:06.307310 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:07:32.995850 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:07:32.995770 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:07:46.286027 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:07:46.285994 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:08:01.899749 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:08:01.896908 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:08:14.988065 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:08:14.988023 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:08:31.327812 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:08:31.327777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:08:47.089743 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:08:47.089703 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:09:04.183548 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:04.183440 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:09:21.498431 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:21.498388 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:09:33.686638 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.686597 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:09:33.690186 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.690143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.693053 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.693018 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ffa3c8\"/\"openshift-service-ca.crt\"" Jul 02 21:09:33.694430 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.694402 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ffa3c8\"/\"default-dockercfg-9swqf\"" Jul 02 21:09:33.694568 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.694435 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ffa3c8\"/\"kube-root-ca.crt\"" Jul 02 21:09:33.694734 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.694717 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ffa3c8\"/\"sub-test-model-ffa3c8-kserve-self-signed-certs\"" Jul 02 21:09:33.700182 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.700156 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:09:33.812365 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.812503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812373 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.812503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812393 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.812503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.812503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812479 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2wj\" (UniqueName: \"kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.812503 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.812502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.912841 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.912972 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.912972 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.912972 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.912972 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2wj\" (UniqueName: \"kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.913192 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.912981 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.913287 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.913267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.913341 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.913318 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.913419 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.913400 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.915361 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.915334 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.915637 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.915616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:33.920710 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:33.920689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2wj\" (UniqueName: \"kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj\") pod \"sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:34.001204 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:34.001153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:34.125650 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:34.125627 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:09:34.128269 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:09:34.128235 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c2b7415_cf9f_4151_929b_be600fab7069.slice/crio-0c5a2583c13c0e4da5abfe19c2acbc48554a98e52ae4e4154bfd55910f564a29 WatchSource:0}: Error finding container 0c5a2583c13c0e4da5abfe19c2acbc48554a98e52ae4e4154bfd55910f564a29: Status 404 returned error can't find the container with id 0c5a2583c13c0e4da5abfe19c2acbc48554a98e52ae4e4154bfd55910f564a29 Jul 02 21:09:34.738818 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:34.738777 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" event={"ID":"8c2b7415-cf9f-4151-929b-be600fab7069","Type":"ContainerStarted","Data":"9e6d6ed31e38b045946a2209ceaa4820735b20a4f2fa9407895a7b24b14a5ec3"} Jul 02 21:09:34.738818 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:34.738825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" event={"ID":"8c2b7415-cf9f-4151-929b-be600fab7069","Type":"ContainerStarted","Data":"0c5a2583c13c0e4da5abfe19c2acbc48554a98e52ae4e4154bfd55910f564a29"} Jul 02 21:09:34.755356 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:34.755303 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" podStartSLOduration=1.755289554 podStartE2EDuration="1.755289554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:09:34.754128913 +0000 UTC m=+2192.623183648" watchObservedRunningTime="2026-07-02 21:09:34.755289554 +0000 UTC m=+2192.624344309" Jul 02 21:09:38.712002 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:38.711961 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:09:44.002032 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:44.001994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:44.002437 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:44.002051 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:44.014341 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:44.014318 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:44.795515 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:44.795488 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:09:55.992442 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:09:55.992405 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:10:15.109925 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:15.109891 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:10:32.386728 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:32.386690 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:10:38.855178 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:38.855146 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:10:49.494873 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:49.494838 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:10:53.874766 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:53.874723 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" podUID="8c2b7415-cf9f-4151-929b-be600fab7069" containerName="main" containerID="cri-o://9e6d6ed31e38b045946a2209ceaa4820735b20a4f2fa9407895a7b24b14a5ec3" gracePeriod=45 Jul 02 21:10:54.071370 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.071342 2569 generic.go:356] "Generic (PLEG): container finished" podID="8c2b7415-cf9f-4151-929b-be600fab7069" containerID="9e6d6ed31e38b045946a2209ceaa4820735b20a4f2fa9407895a7b24b14a5ec3" exitCode=0 Jul 02 21:10:54.071522 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.071412 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" event={"ID":"8c2b7415-cf9f-4151-929b-be600fab7069","Type":"ContainerDied","Data":"9e6d6ed31e38b045946a2209ceaa4820735b20a4f2fa9407895a7b24b14a5ec3"} Jul 02 21:10:54.117387 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.117363 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:10:54.224422 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224422 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224419 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224671 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224500 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224671 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224545 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr2wj\" (UniqueName: \"kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224671 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224569 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224671 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache\") pod \"8c2b7415-cf9f-4151-929b-be600fab7069\" (UID: \"8c2b7415-cf9f-4151-929b-be600fab7069\") " Jul 02 21:10:54.224853 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224728 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 21:10:54.224853 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224755 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home" (OuterVolumeSpecName: "home") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 21:10:54.224962 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224868 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:54.224962 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224916 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:54.224962 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.224941 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 21:10:54.226807 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.226782 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:10:54.226922 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.226813 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm" (OuterVolumeSpecName: "dshm") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 21:10:54.226922 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.226887 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj" (OuterVolumeSpecName: "kube-api-access-qr2wj") pod "8c2b7415-cf9f-4151-929b-be600fab7069" (UID: "8c2b7415-cf9f-4151-929b-be600fab7069"). InnerVolumeSpecName "kube-api-access-qr2wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:10:54.326062 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.326038 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2b7415-cf9f-4151-929b-be600fab7069-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:54.326062 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.326059 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2wj\" (UniqueName: \"kubernetes.io/projected/8c2b7415-cf9f-4151-929b-be600fab7069-kube-api-access-qr2wj\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:54.326214 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.326070 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:54.326214 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:54.326079 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2b7415-cf9f-4151-929b-be600fab7069-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:10:55.077368 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:55.077283 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" event={"ID":"8c2b7415-cf9f-4151-929b-be600fab7069","Type":"ContainerDied","Data":"0c5a2583c13c0e4da5abfe19c2acbc48554a98e52ae4e4154bfd55910f564a29"} Jul 02 21:10:55.077368 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:55.077318 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg" Jul 02 21:10:55.077368 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:55.077339 2569 scope.go:117] "RemoveContainer" containerID="9e6d6ed31e38b045946a2209ceaa4820735b20a4f2fa9407895a7b24b14a5ec3" Jul 02 21:10:55.093482 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:55.093439 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:10:55.098695 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:55.098672 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ffa3c8/sub-test-model-ffa3c8-kserve-6bcff5b5d9-bq9gg"] Jul 02 21:10:56.781799 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:10:56.781764 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2b7415-cf9f-4151-929b-be600fab7069" path="/var/lib/kubelet/pods/8c2b7415-cf9f-4151-929b-be600fab7069/volumes" Jul 02 21:11:35.200622 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:11:35.200585 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:12:00.086282 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:00.086196 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:12:12.719485 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:12.719438 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:12:22.558032 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:22.557999 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:12:22.558530 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:22.558214 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" podUID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" containerName="maas-api" containerID="cri-o://24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e" gracePeriod=30 Jul 02 21:12:22.993323 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:22.993296 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:12:23.164058 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.163997 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca\") pod \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " Jul 02 21:12:23.164058 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.164028 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls\") pod \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " Jul 02 21:12:23.164211 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.164086 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r894d\" (UniqueName: \"kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d\") pod \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\" (UID: \"f71a6ecc-359c-487f-95ab-6e898e54e8ed\") " Jul 02 21:12:23.164389 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.164366 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f71a6ecc-359c-487f-95ab-6e898e54e8ed" (UID: "f71a6ecc-359c-487f-95ab-6e898e54e8ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:12:23.166073 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.166045 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d" (OuterVolumeSpecName: "kube-api-access-r894d") pod "f71a6ecc-359c-487f-95ab-6e898e54e8ed" (UID: "f71a6ecc-359c-487f-95ab-6e898e54e8ed"). InnerVolumeSpecName "kube-api-access-r894d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:12:23.166187 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.166077 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f71a6ecc-359c-487f-95ab-6e898e54e8ed" (UID: "f71a6ecc-359c-487f-95ab-6e898e54e8ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:12:23.264903 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.264878 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71a6ecc-359c-487f-95ab-6e898e54e8ed-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:12:23.264903 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.264900 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71a6ecc-359c-487f-95ab-6e898e54e8ed-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:12:23.265041 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.264913 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r894d\" (UniqueName: \"kubernetes.io/projected/f71a6ecc-359c-487f-95ab-6e898e54e8ed-kube-api-access-r894d\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:12:23.443851 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.443772 2569 generic.go:356] "Generic (PLEG): container finished" podID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" containerID="24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e" exitCode=0 Jul 02 21:12:23.443851 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.443828 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" Jul 02 21:12:23.443999 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.443858 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" event={"ID":"f71a6ecc-359c-487f-95ab-6e898e54e8ed","Type":"ContainerDied","Data":"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e"} Jul 02 21:12:23.443999 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.443914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c" event={"ID":"f71a6ecc-359c-487f-95ab-6e898e54e8ed","Type":"ContainerDied","Data":"f121312578e618954dda8381cebfacf539e7688ffdad99716bc3c893c292f3a2"} Jul 02 21:12:23.443999 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.443938 2569 scope.go:117] "RemoveContainer" containerID="24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e" Jul 02 21:12:23.453231 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.453212 2569 scope.go:117] "RemoveContainer" containerID="24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e" Jul 02 21:12:23.453525 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:12:23.453499 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e\": container with ID starting with 24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e not found: ID does not exist" containerID="24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e" Jul 02 21:12:23.453653 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.453536 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e"} err="failed to get container status \"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e\": rpc error: code = NotFound desc = could not find container \"24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e\": container with ID starting with 24ed35712e4cfe9345abd7a6fb2f4bb1edd8c773a66d9488da03812f05830f5e not found: ID does not exist" Jul 02 21:12:23.465201 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.465176 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:12:23.468491 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:23.468466 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ffa3c8-57f745bcf9-d9c2c"] Jul 02 21:12:24.781676 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:24.781632 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" path="/var/lib/kubelet/pods/f71a6ecc-359c-487f-95ab-6e898e54e8ed/volumes" Jul 02 21:12:26.386889 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:12:26.386841 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:13:05.374338 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.374306 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:13:05.622285 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.622217 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" podUID="cac95532-d903-4e29-8b9a-f0d9df74d843" containerName="maas-api" containerID="cri-o://c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f" gracePeriod=30 Jul 02 21:13:05.861069 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.861047 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:13:05.955863 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.955806 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls\") pod \"cac95532-d903-4e29-8b9a-f0d9df74d843\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " Jul 02 21:13:05.955965 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.955865 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca\") pod \"cac95532-d903-4e29-8b9a-f0d9df74d843\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " Jul 02 21:13:05.955965 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.955882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbxdn\" (UniqueName: \"kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn\") pod \"cac95532-d903-4e29-8b9a-f0d9df74d843\" (UID: \"cac95532-d903-4e29-8b9a-f0d9df74d843\") " Jul 02 21:13:05.956266 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.956239 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cac95532-d903-4e29-8b9a-f0d9df74d843" (UID: "cac95532-d903-4e29-8b9a-f0d9df74d843"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 21:13:05.957857 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.957832 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cac95532-d903-4e29-8b9a-f0d9df74d843" (UID: "cac95532-d903-4e29-8b9a-f0d9df74d843"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 21:13:05.957950 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:05.957915 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn" (OuterVolumeSpecName: "kube-api-access-xbxdn") pod "cac95532-d903-4e29-8b9a-f0d9df74d843" (UID: "cac95532-d903-4e29-8b9a-f0d9df74d843"). InnerVolumeSpecName "kube-api-access-xbxdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 21:13:06.057242 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.057221 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac95532-d903-4e29-8b9a-f0d9df74d843-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:13:06.057242 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.057242 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbxdn\" (UniqueName: \"kubernetes.io/projected/cac95532-d903-4e29-8b9a-f0d9df74d843-kube-api-access-xbxdn\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:13:06.057362 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.057252 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac95532-d903-4e29-8b9a-f0d9df74d843-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Jul 02 21:13:06.628591 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.628557 2569 generic.go:356] "Generic (PLEG): container finished" podID="cac95532-d903-4e29-8b9a-f0d9df74d843" containerID="c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f" exitCode=0 Jul 02 21:13:06.628987 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.628611 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" Jul 02 21:13:06.628987 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.628640 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" event={"ID":"cac95532-d903-4e29-8b9a-f0d9df74d843","Type":"ContainerDied","Data":"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f"} Jul 02 21:13:06.628987 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.628701 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n" event={"ID":"cac95532-d903-4e29-8b9a-f0d9df74d843","Type":"ContainerDied","Data":"a227db7d4769f9493e7c7e61696ed760750786f56b55fd90ce51ddf8cf631f48"} Jul 02 21:13:06.628987 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.628725 2569 scope.go:117] "RemoveContainer" containerID="c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f" Jul 02 21:13:06.638195 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.638177 2569 scope.go:117] "RemoveContainer" containerID="c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f" Jul 02 21:13:06.638490 ip-10-0-140-111 kubenswrapper[2569]: E0702 21:13:06.638445 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f\": container with ID starting with c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f not found: ID does not exist" containerID="c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f" Jul 02 21:13:06.638537 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.638501 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f"} err="failed to get container status \"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f\": rpc error: code = NotFound desc = could not find container \"c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f\": container with ID starting with c5d98322db1e6c57082035103a1b13a5dec80273397cd66b9d9734ea415a5f1f not found: ID does not exist" Jul 02 21:13:06.649463 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.649430 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:13:06.653369 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.653346 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b780eb-7954458cc9-95v5n"] Jul 02 21:13:06.780337 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:06.780312 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac95532-d903-4e29-8b9a-f0d9df74d843" path="/var/lib/kubelet/pods/cac95532-d903-4e29-8b9a-f0d9df74d843/volumes" Jul 02 21:13:08.497179 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:08.497143 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m2dqr"] Jul 02 21:13:17.687862 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:17.687836 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d77d4-74hsh_cea7a512-eecc-4ccd-9db2-b02ab92c267f/copy-manifests/0.log" Jul 02 21:13:17.694620 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:17.694590 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d77d4-74hsh_cea7a512-eecc-4ccd-9db2-b02ab92c267f/manager/0.log" Jul 02 21:13:20.055318 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:20.055285 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m2dqr_cb186070-d25c-453b-8dd7-5f65b13ef2ed/limitador/0.log" Jul 02 21:13:20.675171 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:20.675137 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vs8pr_841e547a-c257-4e5e-9f5b-4cfaa9837ed8/discovery/0.log" Jul 02 21:13:22.109812 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:22.109775 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbs5m9d_74bac2b2-a23c-4803-8878-bb2e841f24d3/main/0.log" Jul 02 21:13:29.373348 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:29.373315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdqwm_74f5fa0b-75e9-4866-afea-d08a6d24e683/global-pull-secret-syncer/0.log" Jul 02 21:13:29.467782 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:29.467747 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jcjrj_6dd58267-4611-45e1-87c2-673d21df0762/konnectivity-agent/0.log" Jul 02 21:13:29.555511 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:29.555486 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-111.ec2.internal_78e29df1a4e6bf20e982f0f14de11bd1/haproxy/0.log" Jul 02 21:13:35.203057 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:35.203022 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m2dqr_cb186070-d25c-453b-8dd7-5f65b13ef2ed/limitador/0.log" Jul 02 21:13:37.054287 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:37.054250 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-6zc8f_907b49ba-ac93-4909-8998-671f55aa7697/cluster-monitoring-operator/0.log" Jul 02 21:13:37.343281 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:37.343180 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74wfr_3f061dd2-5c34-486c-ac8c-9dd94cb51a01/node-exporter/0.log" Jul 02 21:13:37.361173 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:37.361132 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74wfr_3f061dd2-5c34-486c-ac8c-9dd94cb51a01/kube-rbac-proxy/0.log" Jul 02 21:13:37.379901 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:37.379880 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74wfr_3f061dd2-5c34-486c-ac8c-9dd94cb51a01/init-textfile/0.log" Jul 02 21:13:39.447472 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447419 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588"] Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447748 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2b7415-cf9f-4151-929b-be600fab7069" containerName="main" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447772 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2b7415-cf9f-4151-929b-be600fab7069" containerName="main" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447784 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac95532-d903-4e29-8b9a-f0d9df74d843" containerName="maas-api" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447791 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac95532-d903-4e29-8b9a-f0d9df74d843" containerName="maas-api" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447809 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" containerName="maas-api" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447816 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" containerName="maas-api" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447866 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2b7415-cf9f-4151-929b-be600fab7069" containerName="main" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447874 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71a6ecc-359c-487f-95ab-6e898e54e8ed" containerName="maas-api" Jul 02 21:13:39.447878 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.447880 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac95532-d903-4e29-8b9a-f0d9df74d843" containerName="maas-api" Jul 02 21:13:39.450782 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.450756 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.453318 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.453297 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt95b\"/\"openshift-service-ca.crt\"" Jul 02 21:13:39.455047 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.454996 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt95b\"/\"kube-root-ca.crt\"" Jul 02 21:13:39.455166 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.455108 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mt95b\"/\"default-dockercfg-czfgl\"" Jul 02 21:13:39.456640 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.456608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588"] Jul 02 21:13:39.498271 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.498249 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrjf\" (UniqueName: \"kubernetes.io/projected/04fe5978-a8dc-4299-b8dc-e7d60e181c91-kube-api-access-6hrjf\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.498391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.498289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-podres\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.498391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.498309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-sys\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.498391 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.498325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-lib-modules\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.498565 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.498432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-proc\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599659 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrjf\" (UniqueName: \"kubernetes.io/projected/04fe5978-a8dc-4299-b8dc-e7d60e181c91-kube-api-access-6hrjf\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-podres\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-sys\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-lib-modules\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599831 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-proc\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599973 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-sys\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599973 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599846 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-proc\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599973 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599904 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-lib-modules\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.599973 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.599911 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04fe5978-a8dc-4299-b8dc-e7d60e181c91-podres\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.607953 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.607928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrjf\" (UniqueName: \"kubernetes.io/projected/04fe5978-a8dc-4299-b8dc-e7d60e181c91-kube-api-access-6hrjf\") pod \"perf-node-gather-daemonset-kw588\" (UID: \"04fe5978-a8dc-4299-b8dc-e7d60e181c91\") " pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.762529 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.762434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:39.885932 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:39.885898 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588"] Jul 02 21:13:39.889413 ip-10-0-140-111 kubenswrapper[2569]: W0702 21:13:39.889388 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04fe5978_a8dc_4299_b8dc_e7d60e181c91.slice/crio-b3872fd682c0633ca9f1afc586bd169a2bb0e4fa8120230f8cac2acac93d56f0 WatchSource:0}: Error finding container b3872fd682c0633ca9f1afc586bd169a2bb0e4fa8120230f8cac2acac93d56f0: Status 404 returned error can't find the container with id b3872fd682c0633ca9f1afc586bd169a2bb0e4fa8120230f8cac2acac93d56f0 Jul 02 21:13:40.770224 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:40.770185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" event={"ID":"04fe5978-a8dc-4299-b8dc-e7d60e181c91","Type":"ContainerStarted","Data":"d072ce723fd5d055db04523607aee92d623d806b92093569778bf3d3cc001d17"} Jul 02 21:13:40.770659 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:40.770238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" event={"ID":"04fe5978-a8dc-4299-b8dc-e7d60e181c91","Type":"ContainerStarted","Data":"b3872fd682c0633ca9f1afc586bd169a2bb0e4fa8120230f8cac2acac93d56f0"} Jul 02 21:13:40.770659 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:40.770319 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:40.787076 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:40.787025 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" podStartSLOduration=1.787012849 podStartE2EDuration="1.787012849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:13:40.785490523 +0000 UTC m=+2438.654545259" watchObservedRunningTime="2026-07-02 21:13:40.787012849 +0000 UTC m=+2438.656067584" Jul 02 21:13:40.821226 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:40.821203 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-d6mlg_346c2a3b-e613-41b7-addd-b62d91c030ea/volume-data-source-validator/0.log" Jul 02 21:13:41.830104 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:41.830063 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lk8nz_36ac5aff-ebdd-415c-8b7d-46af46993d65/dns/0.log" Jul 02 21:13:41.854185 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:41.854141 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lk8nz_36ac5aff-ebdd-415c-8b7d-46af46993d65/kube-rbac-proxy/0.log" Jul 02 21:13:41.962228 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:41.962204 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tnqkm_7f096e16-c852-477d-bcac-de5ae629e0be/dns-node-resolver/0.log" Jul 02 21:13:42.485302 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:42.485276 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7f85s_a85dfcbb-c1a1-424a-82ec-83c502cd88b3/node-ca/0.log" Jul 02 21:13:43.536237 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:43.536212 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vs8pr_841e547a-c257-4e5e-9f5b-4cfaa9837ed8/discovery/0.log" Jul 02 21:13:44.379263 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:44.379226 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4k6j5_751d0909-edfc-4b48-9019-c83d5baea0cb/serve-healthcheck-canary/0.log" Jul 02 21:13:44.882716 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:44.882691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g7jf_04a93c10-beba-4d5a-bcf3-e7a4fa222f92/kube-rbac-proxy/0.log" Jul 02 21:13:44.901979 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:44.901952 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g7jf_04a93c10-beba-4d5a-bcf3-e7a4fa222f92/exporter/0.log" Jul 02 21:13:44.922838 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:44.922820 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g7jf_04a93c10-beba-4d5a-bcf3-e7a4fa222f92/extractor/0.log" Jul 02 21:13:46.785544 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:46.785507 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mt95b/perf-node-gather-daemonset-kw588" Jul 02 21:13:47.527547 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:47.527514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d77d4-74hsh_cea7a512-eecc-4ccd-9db2-b02ab92c267f/manager/0.log" Jul 02 21:13:47.547721 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:47.547694 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-857d77d4-74hsh_cea7a512-eecc-4ccd-9db2-b02ab92c267f/copy-manifests/0.log" Jul 02 21:13:49.202511 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:49.202428 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d6cb4dc5f-hmvp9_dd6edd46-475d-46f0-8f7a-a62bbfa23709/manager/0.log" Jul 02 21:13:49.226012 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:49.225989 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pd5bf_7fa38dd2-98bb-46ec-9888-2ff983e46fb5/openshift-lws-operator/0.log" Jul 02 21:13:49.716790 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:49.716764 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vs8pr_841e547a-c257-4e5e-9f5b-4cfaa9837ed8/discovery/0.log" Jul 02 21:13:51.093622 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:51.093591 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vs8pr_841e547a-c257-4e5e-9f5b-4cfaa9837ed8/discovery/0.log" Jul 02 21:13:54.061939 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:54.061906 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-mcm2m_9aafc3ae-4262-4207-974f-527fc8915031/migrator/0.log" Jul 02 21:13:54.079425 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:54.079399 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-mcm2m_9aafc3ae-4262-4207-974f-527fc8915031/graceful-termination/0.log" Jul 02 21:13:55.390346 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.390316 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/kube-multus-additional-cni-plugins/0.log" Jul 02 21:13:55.409679 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.409649 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/egress-router-binary-copy/0.log" Jul 02 21:13:55.429661 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.429642 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/cni-plugins/0.log" Jul 02 21:13:55.457966 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.457944 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/bond-cni-plugin/0.log" Jul 02 21:13:55.501900 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.501878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/routeoverride-cni/0.log" Jul 02 21:13:55.556187 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.556165 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/whereabouts-cni-bincopy/0.log" Jul 02 21:13:55.602815 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.602789 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n9d5h_0e7dfe8c-cdb0-4c0e-ad84-647ef1e1637a/whereabouts-cni/0.log" Jul 02 21:13:55.980511 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:55.980489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r7jnz_35cc59a0-b27b-4c94-bed7-fe4459967ed4/kube-multus/0.log" Jul 02 21:13:56.001371 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:56.001348 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hn2fv_a7ca8736-448b-4aeb-bd02-d42cd0f778f4/network-metrics-daemon/0.log" Jul 02 21:13:56.019890 ip-10-0-140-111 kubenswrapper[2569]: I0702 21:13:56.019870 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hn2fv_a7ca8736-448b-4aeb-bd02-d42cd0f778f4/kube-rbac-proxy/0.log"