Jul 09 03:58:22.385978 ip-10-0-141-199 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 03:58:22.385988 ip-10-0-141-199 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 03:58:22.385995 ip-10-0-141-199 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 03:58:22.386197 ip-10-0-141-199 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 03:58:32.412214 ip-10-0-141-199 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 03:58:32.412237 ip-10-0-141-199 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ce0666949b564f8383674cc64a08fe7e -- Jul 09 04:00:12.227654 ip-10-0-141-199 systemd[1]: Starting Kubernetes Kubelet... Jul 09 04:00:12.651889 ip-10-0-141-199 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 04:00:12.651889 ip-10-0-141-199 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 04:00:12.651889 ip-10-0-141-199 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 04:00:12.651889 ip-10-0-141-199 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 04:00:12.651889 ip-10-0-141-199 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 04:00:12.652984 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.652907 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 04:00:12.661156 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661130 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.661156 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661153 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.661156 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661157 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.661156 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661160 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.661156 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661164 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661168 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661171 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661174 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661176 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661179 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661181 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661184 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661187 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661189 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661192 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661194 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661197 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661199 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661201 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661204 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661207 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661212 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661215 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.661619 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661217 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661220 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661222 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661224 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661227 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661229 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661232 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661235 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661237 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661240 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661243 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661245 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661247 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661250 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661254 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661260 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661263 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661268 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661271 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.662109 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661274 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661278 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661280 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661283 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661286 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661288 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661291 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661294 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661298 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661302 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661305 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661308 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661311 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661313 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661315 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661318 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661320 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661323 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661325 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661328 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.662596 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661381 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661384 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661386 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661389 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661392 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661395 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661397 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661400 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661403 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661406 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661409 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661412 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661414 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661417 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661420 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661423 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661426 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661428 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661431 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661434 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.663091 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661437 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661440 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661443 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661446 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661448 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661451 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661453 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661456 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661458 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661461 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661463 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661466 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661468 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661470 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661473 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661475 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.661478 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662167 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662173 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662176 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.663590 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662180 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662183 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662185 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662188 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662191 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662193 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662196 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662198 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662201 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662203 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662206 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662208 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662211 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662213 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662217 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662220 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662222 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662225 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662227 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662230 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.664062 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662232 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662235 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662237 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662239 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662242 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662244 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662247 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662249 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662252 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662254 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662258 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662260 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662264 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662266 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662269 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662271 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662274 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662276 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662279 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662281 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.664570 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662284 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662286 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662289 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662292 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662294 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662296 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662299 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662302 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662304 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662307 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662309 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662312 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662314 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662316 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662319 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662321 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662323 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662326 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662344 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662347 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.665052 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662350 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662352 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662355 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662357 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662361 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662363 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662366 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662368 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662371 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662374 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662376 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662381 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662384 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662387 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662391 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662394 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662397 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662400 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662403 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.665569 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662406 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662409 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662412 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662414 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662417 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662420 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662423 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662425 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662428 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662430 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662433 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662436 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662439 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662442 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662445 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662448 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662451 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662839 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662844 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662847 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.666040 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662850 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662853 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662855 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662858 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662861 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662863 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662866 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662868 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662871 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662873 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662876 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662878 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662881 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662889 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662892 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662894 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662896 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662899 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662901 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662904 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.666542 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662906 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662909 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662911 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662914 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662916 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662918 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662921 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662924 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662927 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662933 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662936 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662939 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662942 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662944 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662947 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662949 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662952 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662954 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662956 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.667029 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662959 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662961 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662963 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662966 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662968 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662971 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662973 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662977 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662979 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662982 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662984 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662987 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662994 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662997 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.662999 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663002 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663004 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663007 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663009 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663012 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.667532 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663015 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663018 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663021 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663024 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663026 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663028 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663031 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663033 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663036 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663038 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663041 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663043 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663046 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663048 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663050 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663053 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663056 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663059 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663061 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.668012 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663063 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663066 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663069 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663073 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663076 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663080 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663083 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663086 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663088 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663090 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663093 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663095 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663098 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663103 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663105 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663108 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663111 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663113 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663196 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663207 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 04:00:12.668526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663217 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663223 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663228 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663233 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663239 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663244 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663248 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663250 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663254 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663257 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663260 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663262 2570 flags.go:64] FLAG: --cgroup-root="" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663265 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663268 2570 flags.go:64] FLAG: --client-ca-file="" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663272 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663275 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663279 2570 flags.go:64] FLAG: --cluster-domain="" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663281 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663284 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663287 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663290 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663293 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663296 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 04:00:12.669003 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663299 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663302 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663307 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663309 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663312 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663316 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663318 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663322 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663325 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663341 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663344 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663347 2570 flags.go:64] FLAG: --enable-server="true" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663350 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663355 2570 flags.go:64] FLAG: --event-burst="100" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663358 2570 flags.go:64] FLAG: --event-qps="50" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663361 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663364 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663367 2570 flags.go:64] FLAG: --eviction-hard="" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663375 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663378 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663381 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663384 2570 flags.go:64] FLAG: --eviction-soft="" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663387 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663389 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663392 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 04:00:12.669601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663395 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663398 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663401 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663404 2570 flags.go:64] FLAG: --feature-gates="" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663407 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663410 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663413 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663416 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663419 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663425 2570 flags.go:64] FLAG: --help="false" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663428 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663431 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663434 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663437 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663440 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663443 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663446 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663449 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663452 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663454 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663457 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663460 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663463 2570 flags.go:64] FLAG: --kube-reserved="" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663465 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 04:00:12.670258 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663468 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663471 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663473 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663476 2570 flags.go:64] FLAG: --lock-file="" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663479 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663494 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663498 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663503 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663510 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663513 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663516 2570 flags.go:64] FLAG: --logging-format="text" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663519 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663522 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663525 2570 flags.go:64] FLAG: --manifest-url="" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663527 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663531 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663534 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663539 2570 flags.go:64] FLAG: --max-pods="110" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663542 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663545 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663547 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663550 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663553 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663556 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663559 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 04:00:12.671540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663566 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663568 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663571 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663574 2570 flags.go:64] FLAG: --pod-cidr="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663577 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663582 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663585 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663587 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663590 2570 flags.go:64] FLAG: --port="10250" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663594 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663596 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0546c9066ef468740" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663600 2570 flags.go:64] FLAG: --qos-reserved="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663602 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663605 2570 flags.go:64] FLAG: --register-node="true" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663608 2570 flags.go:64] FLAG: --register-with-taints="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663611 2570 flags.go:64] FLAG: --registry-burst="10" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663615 2570 flags.go:64] FLAG: --registry-qps="5" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663618 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663620 2570 flags.go:64] FLAG: --reserved-memory="" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663624 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663627 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663630 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663632 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663635 2570 flags.go:64] FLAG: --runonce="false" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663638 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 04:00:12.672170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663641 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663645 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663648 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663651 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663653 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663656 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663659 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663662 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663665 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663667 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663670 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663673 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663675 2570 flags.go:64] FLAG: --system-cgroups="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663678 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663683 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663686 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663689 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663693 2570 flags.go:64] FLAG: --tls-min-version="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663695 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663698 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663701 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663704 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663706 2570 flags.go:64] FLAG: --v="2" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663711 2570 flags.go:64] FLAG: --version="false" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663715 2570 flags.go:64] FLAG: --vmodule="" Jul 09 04:00:12.672851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663718 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.663722 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663832 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663836 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663839 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663842 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663845 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663849 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663852 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663855 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663857 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663860 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663862 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663865 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663868 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663870 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663873 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663875 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663877 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663880 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.673463 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663882 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663885 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663887 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663890 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663892 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663894 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663897 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663899 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663901 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663904 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663906 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663909 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663912 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663914 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663917 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663919 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663922 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663925 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663927 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663930 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.673949 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663934 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663936 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663939 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663941 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663944 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663947 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663949 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663952 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663954 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663958 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663962 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663965 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663967 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663970 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663972 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663975 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663978 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663980 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663983 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.674512 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663985 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663987 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663990 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663992 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663995 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.663997 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664000 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664002 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664005 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664009 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664012 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664015 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664018 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664022 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664025 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664027 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664029 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664032 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664034 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.674983 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664037 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664039 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664042 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664044 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664046 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664049 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664051 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664054 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664057 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664059 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664062 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664065 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664067 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664070 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664072 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664075 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664078 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664080 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664083 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664085 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.675465 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664091 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664095 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.664099 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.664745 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.671243 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.671258 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.671277 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.671286 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671367 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671373 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671376 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671379 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671383 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671387 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.675968 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671390 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671393 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671396 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671398 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671401 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671404 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671406 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671409 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671411 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671414 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671416 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671419 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671421 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671424 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671426 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671429 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671431 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671434 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671437 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.676322 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671440 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671442 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671444 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671447 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671451 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671453 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671456 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671458 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671461 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671463 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671466 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671469 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671471 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671474 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671476 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671479 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671481 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671484 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671486 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671489 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.676832 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671491 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671493 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671496 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671498 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671501 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671504 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671507 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671509 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671511 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671514 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671516 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671519 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671522 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671524 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671527 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671529 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671532 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671537 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671540 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.677307 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671543 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671546 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671549 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671552 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671556 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671559 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671562 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671565 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671568 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671571 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671573 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671575 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671578 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671580 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671583 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671585 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671588 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671590 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671592 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671595 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.677818 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671597 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671600 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671603 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671605 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671608 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671610 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671615 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671618 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671620 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671622 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671625 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671627 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671630 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671632 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671634 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.678353 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.671639 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671748 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671752 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671755 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671758 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671761 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671763 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671766 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671769 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671771 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671773 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671776 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671778 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671780 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671783 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671785 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671788 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671790 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671793 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671795 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 04:00:12.678748 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671798 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671800 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671802 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671805 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671807 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671810 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671813 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671815 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671818 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671820 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671823 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671825 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671827 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671830 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671832 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671835 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671837 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671840 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671844 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671846 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 04:00:12.679234 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671849 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671851 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671854 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671856 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671858 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671861 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671863 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671865 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671868 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671870 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671873 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671875 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671878 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671880 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671882 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671885 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671887 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671890 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671893 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 04:00:12.679747 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671895 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671898 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671900 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671902 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671905 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671907 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671909 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671912 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671914 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671917 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671919 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671921 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671924 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671926 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671928 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671931 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671933 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671936 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671940 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671943 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 04:00:12.680261 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671946 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671949 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671951 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671954 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671957 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671960 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671962 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671965 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671967 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671969 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671972 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671975 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671978 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671980 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671983 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671985 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671987 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671990 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671992 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 04:00:12.680819 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671994 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:12.671997 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.672001 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.672101 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.674135 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.675539 2570 server.go:1019] "Starting client certificate rotation" Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.675634 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.675674 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 04:00:12.681493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.678441 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 04:00:12.681888 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.681867 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 04:00:12.693817 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.693800 2570 log.go:25] "Validated CRI v1 runtime API" Jul 09 04:00:12.696403 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.696390 2570 log.go:25] "Validated CRI v1 image API" Jul 09 04:00:12.697231 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.697213 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 04:00:12.699582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.699553 2570 fs.go:135] Filesystem UUIDs: map[76ac8d63-4088-44ad-bc22-8c3fd25c56a9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e12dcac2-10c6-42c4-bf94-8e3accdf1ba3:/dev/nvme0n1p4] Jul 09 04:00:12.699657 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.699577 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 04:00:12.703432 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.703414 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 04:00:12.704587 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.704472 2570 manager.go:216] Machine: {Timestamp:2026-07-09 04:00:12.703089544 +0000 UTC m=+0.350975654 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121543 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecb97ebb7074ba9e672955ef34436 SystemUUID:ec2ecb97-ebb7-074b-a9e6-72955ef34436 BootID:ce066694-9b56-4f83-8367-4cc64a08fe7e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:3e:3f:08:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:3e:3f:08:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:e6:81:6f:86:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 04:00:12.704587 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.704586 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 04:00:12.704695 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.704660 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 04:00:12.708482 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.708460 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 04:00:12.708612 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.708485 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 04:00:12.709168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.709158 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 04:00:12.709199 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.709169 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 04:00:12.709199 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.709177 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 04:00:12.709827 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.709817 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 04:00:12.711517 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.711507 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 04:00:12.714636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714624 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 09 04:00:12.714795 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714784 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 04:00:12.714843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714816 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 09 04:00:12.714843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714825 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 04:00:12.714843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714839 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 04:00:12.714944 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714848 2570 kubelet.go:398] "Adding apiserver pod source" Jul 09 04:00:12.714944 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.714856 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 04:00:12.715199 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.715188 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 04:00:12.715236 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.715206 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 04:00:12.717954 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.717939 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 04:00:12.721307 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.721283 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 04:00:12.721395 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.721359 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 04:00:12.723391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723375 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723397 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723404 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723410 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723418 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723424 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723432 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 04:00:12.723441 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723438 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 04:00:12.723680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723445 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 04:00:12.723680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723451 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 04:00:12.723680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723460 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 04:00:12.723680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.723466 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 04:00:12.724807 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.724787 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dblxs" Jul 09 04:00:12.725136 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.725125 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 04:00:12.725136 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.725137 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 04:00:12.727083 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.727064 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 04:00:12.727693 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.727675 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 04:00:12.727745 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.727671 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 04:00:12.728216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.728204 2570 server.go:1268] "Started kubelet" Jul 09 04:00:12.728355 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.728287 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 04:00:12.728395 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.728314 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 04:00:12.728513 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.728411 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 04:00:12.729124 ip-10-0-141-199 systemd[1]: Started Kubernetes Kubelet. Jul 09 04:00:12.730105 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.730042 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 04:00:12.730586 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.730563 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 09 04:00:12.732814 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.732793 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dblxs" Jul 09 04:00:12.735162 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.735139 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 04:00:12.737122 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737105 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 04:00:12.737122 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737118 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 04:00:12.737423 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.737365 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:12.737705 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737687 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 04:00:12.737705 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737689 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 04:00:12.737803 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737718 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 04:00:12.737876 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737862 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 09 04:00:12.737928 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.737876 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 09 04:00:12.738144 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738126 2570 factory.go:55] Registering systemd factory Jul 09 04:00:12.738214 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738148 2570 factory.go:223] Registration of the systemd container factory successfully Jul 09 04:00:12.738368 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738354 2570 factory.go:153] Registering CRI-O factory Jul 09 04:00:12.738368 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738369 2570 factory.go:223] Registration of the crio container factory successfully Jul 09 04:00:12.738470 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738418 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 04:00:12.738470 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738443 2570 factory.go:103] Registering Raw factory Jul 09 04:00:12.738470 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.738457 2570 manager.go:1196] Started watching for new ooms in manager Jul 09 04:00:12.739118 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.739105 2570 manager.go:349] Starting recovery of all containers Jul 09 04:00:12.742919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.742900 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:12.746884 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.746863 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-199.ec2.internal\" not found" node="ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.752739 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.752580 2570 manager.go:354] Recovery completed Jul 09 04:00:12.757882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.757867 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.759872 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.759856 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.759923 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.759887 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.759923 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.759901 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.760405 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.760383 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 04:00:12.760405 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.760396 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.760513 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.760396 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 04:00:12.760513 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.760505 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 09 04:00:12.761017 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761002 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.761076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761027 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.761076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761037 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.761823 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761811 2570 policy_none.go:49] "None policy: Start" Jul 09 04:00:12.761862 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761827 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 04:00:12.761862 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.761838 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 04:00:12.762947 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.762936 2570 policy_none.go:47] "Start" Jul 09 04:00:12.804930 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.804916 2570 manager.go:337] "Starting Device Plugin manager" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.804950 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.804973 2570 server.go:87] "Starting device plugin registration server" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.805221 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.805231 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.805326 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.805413 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.805424 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.806007 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 04:00:12.811243 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.806046 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:12.887139 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.887100 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 04:00:12.888541 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.888519 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 04:00:12.888637 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.888555 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 04:00:12.888637 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.888580 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 04:00:12.888637 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.888611 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 04:00:12.891038 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.891014 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:12.906262 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.906191 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.907141 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.907120 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.907227 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.907157 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.907227 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.907169 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.907227 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.907191 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.915999 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.915981 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.916064 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.916010 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-199.ec2.internal\": node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:12.931505 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:12.931481 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:12.988951 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.988886 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal"] Jul 09 04:00:12.989106 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.988992 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.990504 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.990486 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.990580 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.990524 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.990580 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.990536 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.991895 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.991883 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.991991 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.991979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.992028 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992005 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.992627 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992604 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.992712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992635 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.992712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992648 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.992712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992668 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.992712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992690 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.992712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.992704 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:12.993886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.993871 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:12.993941 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.993902 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 04:00:12.994593 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.994564 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientMemory" Jul 09 04:00:12.994593 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.994587 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 04:00:12.994725 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:12.994609 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeHasSufficientPID" Jul 09 04:00:13.014940 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.014920 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-199.ec2.internal\" not found" node="ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.019243 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.019226 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-199.ec2.internal\" not found" node="ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.032420 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.032398 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.040453 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.040423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.040563 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.040454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91918a83cd21a5c27f7945240cf37d77-config\") pod \"kube-apiserver-proxy-ip-10-0-141-199.ec2.internal\" (UID: \"91918a83cd21a5c27f7945240cf37d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.040563 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.040478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.133303 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.133250 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.141638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141597 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91918a83cd21a5c27f7945240cf37d77-config\") pod \"kube-apiserver-proxy-ip-10-0-141-199.ec2.internal\" (UID: \"91918a83cd21a5c27f7945240cf37d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.141638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91918a83cd21a5c27f7945240cf37d77-config\") pod \"kube-apiserver-proxy-ip-10-0-141-199.ec2.internal\" (UID: \"91918a83cd21a5c27f7945240cf37d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.141817 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.141817 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.141817 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141720 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.141817 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.141745 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8eb13dd227d9300d586f0f80eae159-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal\" (UID: \"7c8eb13dd227d9300d586f0f80eae159\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.234132 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.234039 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.318575 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.318541 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.322520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.322385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.335032 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.335010 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.435575 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.435542 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.536141 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.536069 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.636706 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.636680 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-199.ec2.internal\" not found" Jul 09 04:00:13.652605 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.652583 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:13.674649 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.674624 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 04:00:13.674788 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.674769 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:13.674828 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.674777 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:13.674828 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.674791 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 04:00:13.715969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.715937 2570 apiserver.go:52] "Watching apiserver" Jul 09 04:00:13.737273 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737250 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 04:00:13.737273 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737271 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 04:00:13.737603 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737585 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.737690 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737580 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 03:55:14 +0000 UTC" deadline="2028-01-30 17:59:34.363552236 +0000 UTC" Jul 09 04:00:13.737744 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737696 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13693h59m20.625860846s" Jul 09 04:00:13.737877 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.737859 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-pr2n4","openshift-cluster-node-tuning-operator/tuned-6b6gw","openshift-dns/node-resolver-zq56v","openshift-multus/multus-drpsr","openshift-network-diagnostics/network-check-target-fccbg","openshift-network-operator/iptables-alerter-6rkgx","kube-system/global-pull-secret-syncer-z872w","kube-system/konnectivity-agent-lm56f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62","openshift-image-registry/node-ca-gzh4j","openshift-multus/multus-additional-cni-plugins-82x8f","openshift-multus/network-metrics-daemon-r64cg"] Jul 09 04:00:13.739671 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.739653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.741296 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.741276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.744135 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.742526 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 04:00:13.744135 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.743624 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-928c7\"" Jul 09 04:00:13.744135 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.743873 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 04:00:13.744135 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744014 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.744399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744205 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.744399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744257 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 04:00:13.744399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744258 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.744399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744271 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 04:00:13.744399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744391 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.744615 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.744509 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.745210 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.745186 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ssm2l\"" Jul 09 04:00:13.746753 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.746728 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.746753 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.746749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d88sv\"" Jul 09 04:00:13.746909 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.746879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.747256 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.747242 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.748000 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.747986 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 04:00:13.748074 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.748058 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" Jul 09 04:00:13.748126 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.748117 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:13.748215 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.748194 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:13.749281 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749267 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xgrpj\"" Jul 09 04:00:13.749393 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749379 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.749504 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749489 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.749561 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749489 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 04:00:13.749610 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749596 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 04:00:13.749843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.749826 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.750516 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.750500 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 04:00:13.750612 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.750599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.750699 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.750672 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:13.751935 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.751922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.752305 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.752284 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.752416 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.752408 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8q8zp\"" Jul 09 04:00:13.752518 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.752505 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.752740 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.752723 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 04:00:13.753683 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.753666 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.754435 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.754415 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 04:00:13.754532 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.754484 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x8fpb\"" Jul 09 04:00:13.754588 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.754560 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 04:00:13.755271 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.755251 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.756699 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.756683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.756812 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.756792 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.756878 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.756864 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.757478 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.757463 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ftz8r\"" Jul 09 04:00:13.757562 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.757490 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 04:00:13.757562 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.757543 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 04:00:13.758105 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.758090 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wxmsm\"" Jul 09 04:00:13.758105 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.758104 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 04:00:13.758216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.758110 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 04:00:13.758972 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.758957 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal"] Jul 09 04:00:13.759042 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.759030 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.759090 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.759076 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:13.759227 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.759211 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 04:00:13.759359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.759314 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7jbbn\"" Jul 09 04:00:13.759458 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.759444 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 04:00:13.759510 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.759450 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal"] Jul 09 04:00:13.760364 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.760351 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 04:00:13.783989 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.783963 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p66d9" Jul 09 04:00:13.793760 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.793742 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p66d9" Jul 09 04:00:13.839554 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.839531 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 04:00:13.845377 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb69976-0b5d-41c7-97de-84e379767d58-tmp-dir\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.845377 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-kubelet\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.845377 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-sys-fs\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.845553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-iptables-alerter-script\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.845553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-host-slash\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.845553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djx9\" (UniqueName: \"kubernetes.io/projected/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-kube-api-access-8djx9\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.845553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.845553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-host\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-conf-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-registration-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b2ff288-75bd-49e2-a498-146c506d2b02-ovn-node-metrics-cert\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cnibin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-os-release\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845749 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.845780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-slash\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845792 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-log-socket\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-systemd\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f46\" (UniqueName: \"kubernetes.io/projected/c9cdfb90-1b95-4111-815e-163feaf59d6c-kube-api-access-b6f46\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnm6\" (UniqueName: \"kubernetes.io/projected/eeba12f8-f6d5-413f-b135-50720a67ab29-kube-api-access-7xnm6\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-netns\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-script-lib\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-tuned\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-netns\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.845997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-device-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wjj\" (UniqueName: \"kubernetes.io/projected/e3a6e221-92b6-4128-ae09-a15961f44acf-kube-api-access-d7wjj\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-kubelet-config\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-run\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-k8s-cni-cncf-io\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846205 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-multus-certs\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-binary-copy\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-systemd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-conf\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-system-cni-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-node-log\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-env-overrides\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-kubernetes\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-lib-modules\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-var-lib-kubelet\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcb69976-0b5d-41c7-97de-84e379767d58-hosts-file\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295s8\" (UniqueName: \"kubernetes.io/projected/bcb69976-0b5d-41c7-97de-84e379767d58-kube-api-access-295s8\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87c8df8-b0b7-417f-bf10-932250ae18cc-agent-certs\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-socket-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-socket-dir-parent\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-hostroot\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.846762 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-etc-kubernetes\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-dbus\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zl5\" (UniqueName: \"kubernetes.io/projected/04568c63-abf8-4625-a487-478ab797e339-kube-api-access-d5zl5\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-os-release\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-kubelet\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-bin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-bin\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvx9\" (UniqueName: \"kubernetes.io/projected/4b2ff288-75bd-49e2-a498-146c506d2b02-kube-api-access-9dvx9\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zdc\" (UniqueName: \"kubernetes.io/projected/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-kube-api-access-56zdc\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.846983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87c8df8-b0b7-417f-bf10-932250ae18cc-konnectivity-ca\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-cnibin\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-systemd-units\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-ovn\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-tmp\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cni-binary-copy\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-multus\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-daemon-config\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-netd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsc7\" (UniqueName: \"kubernetes.io/projected/8090bdf4-63af-4af9-949d-2453f457d4da-kube-api-access-8zsc7\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04568c63-abf8-4625-a487-478ab797e339-host\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847316 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04568c63-abf8-4625-a487-478ab797e339-serviceca\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-etc-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-config\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-sys\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-var-lib-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-modprobe-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.847857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-system-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.848359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.847482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysconfig\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.904779 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:13.904746 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8eb13dd227d9300d586f0f80eae159.slice/crio-98f75464b8ded4e74e4702e6be2e9301fc861e1bd285d4a8ed8dc7a295e43793 WatchSource:0}: Error finding container 98f75464b8ded4e74e4702e6be2e9301fc861e1bd285d4a8ed8dc7a295e43793: Status 404 returned error can't find the container with id 98f75464b8ded4e74e4702e6be2e9301fc861e1bd285d4a8ed8dc7a295e43793 Jul 09 04:00:13.909119 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.909102 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:00:13.947856 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-slash\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-log-socket\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-systemd\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f46\" (UniqueName: \"kubernetes.io/projected/c9cdfb90-1b95-4111-815e-163feaf59d6c-kube-api-access-b6f46\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnm6\" (UniqueName: \"kubernetes.io/projected/eeba12f8-f6d5-413f-b135-50720a67ab29-kube-api-access-7xnm6\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947952 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-slash\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-netns\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.947971 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-log-socket\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.947994 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-script-lib\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948011 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-systemd\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.948047 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-tuned\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-netns\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-netns\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-device-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wjj\" (UniqueName: \"kubernetes.io/projected/e3a6e221-92b6-4128-ae09-a15961f44acf-kube-api-access-d7wjj\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.948165 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:14.44813667 +0000 UTC m=+2.096022766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948210 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-netns\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-kubelet-config\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-run\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-k8s-cni-cncf-io\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948275 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-multus-certs\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-binary-copy\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-systemd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948360 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-kubelet-config\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.948506 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-conf\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-system-cni-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948393 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-k8s-cni-cncf-io\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-node-log\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-env-overrides\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-kubernetes\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948434 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-run\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948437 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-lib-modules\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948270 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-device-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-var-lib-kubelet\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948473 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-systemd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948483 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-node-log\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcb69976-0b5d-41c7-97de-84e379767d58-hosts-file\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295s8\" (UniqueName: \"kubernetes.io/projected/bcb69976-0b5d-41c7-97de-84e379767d58-kube-api-access-295s8\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948524 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcb69976-0b5d-41c7-97de-84e379767d58-hosts-file\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87c8df8-b0b7-417f-bf10-932250ae18cc-agent-certs\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-conf\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-socket-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-socket-dir-parent\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948610 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-run-multus-certs\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-system-cni-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-hostroot\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-etc-kubernetes\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948717 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-dbus\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zl5\" (UniqueName: \"kubernetes.io/projected/04568c63-abf8-4625-a487-478ab797e339-kube-api-access-d5zl5\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-os-release\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948794 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-script-lib\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948796 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-env-overrides\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948844 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-hostroot\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948873 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-kubernetes\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948882 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-kubelet\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948915 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-lib-modules\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-kubelet\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948932 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.949919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948932 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-bin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948993 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-os-release\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.948998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-bin\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949010 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-binary-copy\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949034 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-var-lib-kubelet\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949044 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-bin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949061 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949074 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-bin\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949070 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvx9\" (UniqueName: \"kubernetes.io/projected/4b2ff288-75bd-49e2-a498-146c506d2b02-kube-api-access-9dvx9\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949094 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb7a001d-b4c3-4aba-a69e-786b015d507a-dbus\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zdc\" (UniqueName: \"kubernetes.io/projected/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-kube-api-access-56zdc\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949141 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-etc-kubernetes\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87c8df8-b0b7-417f-bf10-932250ae18cc-konnectivity-ca\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949165 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-socket-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.950764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-cnibin\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-systemd-units\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-ovn\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949240 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-socket-dir-parent\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-systemd-units\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-tmp\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949292 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-run-ovn\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cni-binary-copy\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949222 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysctl-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949292 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-cnibin\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-multus\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-daemon-config\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-netd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949438 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-cni-multus\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsc7\" (UniqueName: \"kubernetes.io/projected/8090bdf4-63af-4af9-949d-2453f457d4da-kube-api-access-8zsc7\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949485 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:13.951451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949499 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04568c63-abf8-4625-a487-478ab797e339-host\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04568c63-abf8-4625-a487-478ab797e339-serviceca\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949559 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04568c63-abf8-4625-a487-478ab797e339-host\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-etc-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-config\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-sys\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-var-lib-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-modprobe-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949716 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-system-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysconfig\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949747 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87c8df8-b0b7-417f-bf10-932250ae18cc-konnectivity-ca\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb69976-0b5d-41c7-97de-84e379767d58-tmp-dir\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-kubelet\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949829 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-sys-fs\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-iptables-alerter-script\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-host-slash\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.951969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djx9\" (UniqueName: \"kubernetes.io/projected/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-kube-api-access-8djx9\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949917 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cni-binary-copy\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-host\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-conf-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.949990 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-daemon-config\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-registration-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b2ff288-75bd-49e2-a498-146c506d2b02-ovn-node-metrics-cert\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cnibin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-os-release\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950147 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-host-cni-netd\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b2ff288-75bd-49e2-a498-146c506d2b02-ovnkube-config\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950198 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-sys\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-host-var-lib-kubelet\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.952468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950256 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-sysconfig\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950278 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeba12f8-f6d5-413f-b135-50720a67ab29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950301 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-system-cni-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950327 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-host-slash\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950377 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-registration-dir\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.950404 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950445 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-etc-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950507 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-os-release\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.950536 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:14.450504249 +0000 UTC m=+2.098390348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950544 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-multus-conf-dir\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950616 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-modprobe-d\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950618 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb69976-0b5d-41c7-97de-84e379767d58-tmp-dir\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950659 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9cdfb90-1b95-4111-815e-163feaf59d6c-host\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950673 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-cnibin\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950705 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b2ff288-75bd-49e2-a498-146c506d2b02-var-lib-openvswitch\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950795 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950838 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8090bdf4-63af-4af9-949d-2453f457d4da-sys-fs\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.953090 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950884 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.950984 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04568c63-abf8-4625-a487-478ab797e339-serviceca\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.951170 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-iptables-alerter-script\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.951307 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eeba12f8-f6d5-413f-b135-50720a67ab29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.953263 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-etc-tuned\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.953303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9cdfb90-1b95-4111-815e-163feaf59d6c-tmp\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.953307 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b2ff288-75bd-49e2-a498-146c506d2b02-ovn-node-metrics-cert\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:13.953680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.953429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87c8df8-b0b7-417f-bf10-932250ae18cc-agent-certs\") pod \"konnectivity-agent-lm56f\" (UID: \"b87c8df8-b0b7-417f-bf10-932250ae18cc\") " pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:13.959909 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.959881 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:13.959909 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.959914 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:13.960083 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.959928 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:13.960083 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:13.959990 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:14.459970983 +0000 UTC m=+2.107857131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:13.960531 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.960501 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnm6\" (UniqueName: \"kubernetes.io/projected/eeba12f8-f6d5-413f-b135-50720a67ab29-kube-api-access-7xnm6\") pod \"multus-additional-cni-plugins-82x8f\" (UID: \"eeba12f8-f6d5-413f-b135-50720a67ab29\") " pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:13.960778 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.960753 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295s8\" (UniqueName: \"kubernetes.io/projected/bcb69976-0b5d-41c7-97de-84e379767d58-kube-api-access-295s8\") pod \"node-resolver-zq56v\" (UID: \"bcb69976-0b5d-41c7-97de-84e379767d58\") " pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:13.961390 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.961361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wjj\" (UniqueName: \"kubernetes.io/projected/e3a6e221-92b6-4128-ae09-a15961f44acf-kube-api-access-d7wjj\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:13.962224 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.961769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f46\" (UniqueName: \"kubernetes.io/projected/c9cdfb90-1b95-4111-815e-163feaf59d6c-kube-api-access-b6f46\") pod \"tuned-6b6gw\" (UID: \"c9cdfb90-1b95-4111-815e-163feaf59d6c\") " pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:13.962224 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.962210 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsc7\" (UniqueName: \"kubernetes.io/projected/8090bdf4-63af-4af9-949d-2453f457d4da-kube-api-access-8zsc7\") pod \"aws-ebs-csi-driver-node-l6k62\" (UID: \"8090bdf4-63af-4af9-949d-2453f457d4da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:13.962554 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.962535 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zdc\" (UniqueName: \"kubernetes.io/projected/ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec-kube-api-access-56zdc\") pod \"multus-drpsr\" (UID: \"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec\") " pod="openshift-multus/multus-drpsr" Jul 09 04:00:13.962629 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.962607 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djx9\" (UniqueName: \"kubernetes.io/projected/0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a-kube-api-access-8djx9\") pod \"iptables-alerter-6rkgx\" (UID: \"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a\") " pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:13.962727 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.962709 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zl5\" (UniqueName: \"kubernetes.io/projected/04568c63-abf8-4625-a487-478ab797e339-kube-api-access-d5zl5\") pod \"node-ca-gzh4j\" (UID: \"04568c63-abf8-4625-a487-478ab797e339\") " pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:13.963001 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:13.962987 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvx9\" (UniqueName: \"kubernetes.io/projected/4b2ff288-75bd-49e2-a498-146c506d2b02-kube-api-access-9dvx9\") pod \"ovnkube-node-pr2n4\" (UID: \"4b2ff288-75bd-49e2-a498-146c506d2b02\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:15.603373 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.603278 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:15.614072 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.614026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2ff288_75bd_49e2_a498_146c506d2b02.slice/crio-01bd97065817cb4cdb0f825ae11d88585edb32a6f79be8c633c94afb2d16ee37 WatchSource:0}: Error finding container 01bd97065817cb4cdb0f825ae11d88585edb32a6f79be8c633c94afb2d16ee37: Status 404 returned error can't find the container with id 01bd97065817cb4cdb0f825ae11d88585edb32a6f79be8c633c94afb2d16ee37 Jul 09 04:00:15.619267 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.619247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" Jul 09 04:00:15.626562 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.626536 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cdfb90_1b95_4111_815e_163feaf59d6c.slice/crio-669e56ce14a9339b5afbabd5f2309c24a235f5a4210f0237d8a114be5f5e6c21 WatchSource:0}: Error finding container 669e56ce14a9339b5afbabd5f2309c24a235f5a4210f0237d8a114be5f5e6c21: Status 404 returned error can't find the container with id 669e56ce14a9339b5afbabd5f2309c24a235f5a4210f0237d8a114be5f5e6c21 Jul 09 04:00:15.626643 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.626575 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zq56v" Jul 09 04:00:15.628709 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.628645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drpsr" Jul 09 04:00:15.636126 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.636106 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb69976_0b5d_41c7_97de_84e379767d58.slice/crio-79690f55b3de66023dafab08f4d0a6e965d939dfcd061509b1dd98c291110bf3 WatchSource:0}: Error finding container 79690f55b3de66023dafab08f4d0a6e965d939dfcd061509b1dd98c291110bf3: Status 404 returned error can't find the container with id 79690f55b3de66023dafab08f4d0a6e965d939dfcd061509b1dd98c291110bf3 Jul 09 04:00:15.636541 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.636520 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8cd9ab_b1e1_401e_ab33_c3c46a28d8ec.slice/crio-44ffab87a10aeab00a41c63ffcbe61c8dcdf1ea14b1a5860bb11f0d42f95cdbb WatchSource:0}: Error finding container 44ffab87a10aeab00a41c63ffcbe61c8dcdf1ea14b1a5860bb11f0d42f95cdbb: Status 404 returned error can't find the container with id 44ffab87a10aeab00a41c63ffcbe61c8dcdf1ea14b1a5860bb11f0d42f95cdbb Jul 09 04:00:15.659082 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.659015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6rkgx" Jul 09 04:00:15.667167 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.667136 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3a6cfb_3551_4d0b_8e42_7c28306cfe5a.slice/crio-9d272e8e2b43ac4c9b0a5b477d049bdb46161e91600c0118d2ed7edc8f468b90 WatchSource:0}: Error finding container 9d272e8e2b43ac4c9b0a5b477d049bdb46161e91600c0118d2ed7edc8f468b90: Status 404 returned error can't find the container with id 9d272e8e2b43ac4c9b0a5b477d049bdb46161e91600c0118d2ed7edc8f468b90 Jul 09 04:00:15.672139 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.672119 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:15.674191 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.674173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:15.682163 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.682141 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87c8df8_b0b7_417f_bf10_932250ae18cc.slice/crio-bd5e8e679342ebce9ae0919272af534b9a3e750d13afc89c284e5f4800f99288 WatchSource:0}: Error finding container bd5e8e679342ebce9ae0919272af534b9a3e750d13afc89c284e5f4800f99288: Status 404 returned error can't find the container with id bd5e8e679342ebce9ae0919272af534b9a3e750d13afc89c284e5f4800f99288 Jul 09 04:00:15.692369 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.692349 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" Jul 09 04:00:15.699026 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.699000 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8090bdf4_63af_4af9_949d_2453f457d4da.slice/crio-1ef46028eede8449f956cffd94ec8af1f3a5f192b623ad4fa554e181a8c88782 WatchSource:0}: Error finding container 1ef46028eede8449f956cffd94ec8af1f3a5f192b623ad4fa554e181a8c88782: Status 404 returned error can't find the container with id 1ef46028eede8449f956cffd94ec8af1f3a5f192b623ad4fa554e181a8c88782 Jul 09 04:00:15.713098 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.713067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzh4j" Jul 09 04:00:15.718185 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.718158 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82x8f" Jul 09 04:00:15.721105 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.721085 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04568c63_abf8_4625_a487_478ab797e339.slice/crio-4f7877370e71c3236cd8934a7e7531d96ba2c53005fc10112a453432d9592d77 WatchSource:0}: Error finding container 4f7877370e71c3236cd8934a7e7531d96ba2c53005fc10112a453432d9592d77: Status 404 returned error can't find the container with id 4f7877370e71c3236cd8934a7e7531d96ba2c53005fc10112a453432d9592d77 Jul 09 04:00:15.725379 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:15.725355 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeba12f8_f6d5_413f_b135_50720a67ab29.slice/crio-95decbb9f2acc7bdc9e1502f92a052c7bdc1686228876ffddfaa002e47e1ca1a WatchSource:0}: Error finding container 95decbb9f2acc7bdc9e1502f92a052c7bdc1686228876ffddfaa002e47e1ca1a: Status 404 returned error can't find the container with id 95decbb9f2acc7bdc9e1502f92a052c7bdc1686228876ffddfaa002e47e1ca1a Jul 09 04:00:15.986369 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.986335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:15.986552 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:15.986377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:15.986552 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:15.986482 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:15.986552 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:15.986525 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:15.986552 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:15.986548 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:16.98652883 +0000 UTC m=+3.101705844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:15.986747 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:15.986575 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:16.98655937 +0000 UTC m=+3.101736399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:16.021961 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:16.021918 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91918a83cd21a5c27f7945240cf37d77.slice/crio-fd35df147e019e397c773efb12b3ad5fbe8e5dcb62ccf1cdcc17618ba74b97ae WatchSource:0}: Error finding container fd35df147e019e397c773efb12b3ad5fbe8e5dcb62ccf1cdcc17618ba74b97ae: Status 404 returned error can't find the container with id fd35df147e019e397c773efb12b3ad5fbe8e5dcb62ccf1cdcc17618ba74b97ae Jul 09 04:00:16.087211 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.087178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:16.087389 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.087347 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:16.087389 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.087367 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:16.087389 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.087377 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:16.087554 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.087431 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:17.087416074 +0000 UTC m=+3.202593092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:16.328748 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.328632 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:55:15 +0000 UTC" deadline="2028-04-17 19:36:09.108367976 +0000 UTC" Jul 09 04:00:16.328748 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.328703 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15567h35m52.779672221s" Jul 09 04:00:16.434461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.434403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6rkgx" event={"ID":"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a","Type":"ContainerStarted","Data":"9d272e8e2b43ac4c9b0a5b477d049bdb46161e91600c0118d2ed7edc8f468b90"} Jul 09 04:00:16.436804 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.436757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drpsr" event={"ID":"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec","Type":"ContainerStarted","Data":"44ffab87a10aeab00a41c63ffcbe61c8dcdf1ea14b1a5860bb11f0d42f95cdbb"} Jul 09 04:00:16.439576 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.439509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zq56v" event={"ID":"bcb69976-0b5d-41c7-97de-84e379767d58","Type":"ContainerStarted","Data":"79690f55b3de66023dafab08f4d0a6e965d939dfcd061509b1dd98c291110bf3"} Jul 09 04:00:16.441389 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.441325 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" event={"ID":"c9cdfb90-1b95-4111-815e-163feaf59d6c","Type":"ContainerStarted","Data":"669e56ce14a9339b5afbabd5f2309c24a235f5a4210f0237d8a114be5f5e6c21"} Jul 09 04:00:16.443019 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.442955 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" event={"ID":"91918a83cd21a5c27f7945240cf37d77","Type":"ContainerStarted","Data":"fd35df147e019e397c773efb12b3ad5fbe8e5dcb62ccf1cdcc17618ba74b97ae"} Jul 09 04:00:16.452996 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.452896 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerStarted","Data":"95decbb9f2acc7bdc9e1502f92a052c7bdc1686228876ffddfaa002e47e1ca1a"} Jul 09 04:00:16.454695 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.454617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lm56f" event={"ID":"b87c8df8-b0b7-417f-bf10-932250ae18cc","Type":"ContainerStarted","Data":"bd5e8e679342ebce9ae0919272af534b9a3e750d13afc89c284e5f4800f99288"} Jul 09 04:00:16.456332 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.456244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"01bd97065817cb4cdb0f825ae11d88585edb32a6f79be8c633c94afb2d16ee37"} Jul 09 04:00:16.458138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.458077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" event={"ID":"7c8eb13dd227d9300d586f0f80eae159","Type":"ContainerStarted","Data":"98f75464b8ded4e74e4702e6be2e9301fc861e1bd285d4a8ed8dc7a295e43793"} Jul 09 04:00:16.459544 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.459479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzh4j" event={"ID":"04568c63-abf8-4625-a487-478ab797e339","Type":"ContainerStarted","Data":"4f7877370e71c3236cd8934a7e7531d96ba2c53005fc10112a453432d9592d77"} Jul 09 04:00:16.462019 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.461994 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" event={"ID":"8090bdf4-63af-4af9-949d-2453f457d4da","Type":"ContainerStarted","Data":"1ef46028eede8449f956cffd94ec8af1f3a5f192b623ad4fa554e181a8c88782"} Jul 09 04:00:16.548137 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.548103 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:16.609444 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.609367 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:16.996202 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.996166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:16.996389 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:16.996221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:16.996389 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.996362 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:16.996511 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.996425 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:18.996404119 +0000 UTC m=+5.111581145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:16.996827 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.996808 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:16.996918 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:16.996861 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:18.996845758 +0000 UTC m=+5.112022778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:17.097856 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.097819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:17.098055 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.097993 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:17.098055 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.098012 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:17.098055 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.098025 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:17.098281 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.098103 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:19.098081786 +0000 UTC m=+5.213258806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:17.329241 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.329113 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:55:15 +0000 UTC" deadline="2028-01-17 15:22:53.107132032 +0000 UTC" Jul 09 04:00:17.329241 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.329148 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13379h22m35.777988969s" Jul 09 04:00:17.421692 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.421458 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:17.421692 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.421497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:17.421692 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.421591 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:17.421967 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.421704 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:17.421967 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.421456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:17.421967 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:17.421795 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:17.933871 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:17.933644 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 04:00:18.473613 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:18.473567 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" event={"ID":"7c8eb13dd227d9300d586f0f80eae159","Type":"ContainerStarted","Data":"ae09928bb624a6b1120f457ab05261a4548efb80ad0633d7fef6095f2dc66bb8"} Jul 09 04:00:19.017364 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.017321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:19.017547 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.017387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:19.017622 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.017565 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:19.017682 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.017630 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:23.017610248 +0000 UTC m=+9.132787276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:19.018022 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.018004 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:19.018152 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.018076 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:23.018060116 +0000 UTC m=+9.133237132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:19.118532 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.118490 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:19.118725 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.118651 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:19.118725 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.118672 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:19.118725 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.118685 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:19.118890 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.118742 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:23.118723698 +0000 UTC m=+9.233900718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:19.422294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.422259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:19.422487 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.422408 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:19.422840 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.422818 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:19.422933 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.422914 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:19.422989 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:19.422978 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:19.423095 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:19.423076 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:21.422526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:21.422489 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:21.422976 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:21.422615 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:21.423070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:21.422998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:21.423321 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:21.423134 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:21.423321 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:21.423190 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:21.423321 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:21.423257 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:23.048877 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.048832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:23.049363 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.048899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:23.049363 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.049084 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:23.049363 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.049150 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:31.049129845 +0000 UTC m=+17.164306874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:23.049593 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.049575 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:23.049671 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.049633 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:31.049616915 +0000 UTC m=+17.164793941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:23.149795 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.149719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:23.149949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.149890 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:23.149949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.149906 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:23.149949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.149916 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:23.150134 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.149965 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:31.149951395 +0000 UTC m=+17.265128428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:23.422594 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.422550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:23.422741 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.422691 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:23.423105 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.423081 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:23.423231 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.423181 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:23.423231 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:23.423191 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:23.423339 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:23.423309 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:25.422521 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:25.422488 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:25.422963 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:25.422596 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:25.422963 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:25.422867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:25.422963 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:25.422927 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:25.422963 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:25.422960 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:25.423198 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:25.423008 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:25.491522 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:25.491486 2570 generic.go:356] "Generic (PLEG): container finished" podID="7c8eb13dd227d9300d586f0f80eae159" containerID="ae09928bb624a6b1120f457ab05261a4548efb80ad0633d7fef6095f2dc66bb8" exitCode=0 Jul 09 04:00:25.491683 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:25.491586 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" event={"ID":"7c8eb13dd227d9300d586f0f80eae159","Type":"ContainerDied","Data":"ae09928bb624a6b1120f457ab05261a4548efb80ad0633d7fef6095f2dc66bb8"} Jul 09 04:00:27.422499 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:27.422463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:27.422949 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:27.422475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:27.422949 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:27.422484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:27.422949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:27.422598 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:27.422949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:27.422721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:27.422949 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:27.422757 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:29.422507 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:29.422470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:29.422967 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:29.422529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:29.422967 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:29.422547 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:29.422967 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:29.422656 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:29.423156 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:29.423121 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:29.423234 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:29.423206 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:31.114059 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.114004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:31.114549 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.114074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:31.114549 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.114166 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:31.114549 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.114168 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:31.114549 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.114220 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:00:47.114205626 +0000 UTC m=+33.229382640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:31.114549 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.114233 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:47.114227645 +0000 UTC m=+33.229404659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:31.214951 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.214363 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:31.214951 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.214538 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:31.214951 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.214559 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:31.214951 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.214572 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:31.214951 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.214619 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:47.21460186 +0000 UTC m=+33.329778878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:31.421862 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.421833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:31.422014 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.421937 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:31.422014 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.421941 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:31.422014 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:31.421963 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:31.422159 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.422037 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:31.422159 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:31.422118 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:33.422498 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:33.422457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:33.422971 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:33.422457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:33.422971 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:33.422589 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:33.422971 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:33.422457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:33.422971 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:33.422661 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:33.422971 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:33.422742 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:35.421517 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:35.421478 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:35.421987 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:35.421523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:35.421987 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:35.421621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:35.421987 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:35.421634 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:35.421987 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:35.421718 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:35.421987 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:35.421776 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:36.514150 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.513931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drpsr" event={"ID":"ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec","Type":"ContainerStarted","Data":"b1a89ef75eb6b77d4e82b5ae8305ccdd73ec0ba04a8047d75463033066ca388e"} Jul 09 04:00:36.515538 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.515510 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zq56v" event={"ID":"bcb69976-0b5d-41c7-97de-84e379767d58","Type":"ContainerStarted","Data":"cb23bf4f4d5cf49829ca5eb9ae210dd480be9afaccf0ebc21cba81accddf7cf6"} Jul 09 04:00:36.516988 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.516963 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" event={"ID":"c9cdfb90-1b95-4111-815e-163feaf59d6c","Type":"ContainerStarted","Data":"527a7b653adf05ff95db9a5c037090200a1c7a2c7a619c04bf72abd72198bb91"} Jul 09 04:00:36.518472 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.518445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" event={"ID":"91918a83cd21a5c27f7945240cf37d77","Type":"ContainerStarted","Data":"4010c5b1a777ed6819b38a237d29f9e24b4d331dd8ec53429d9a7980f51d7993"} Jul 09 04:00:36.519752 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.519730 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="4336b78fe521beb28337182202dd2952dabefcf0c332580a97fbf47870d666df" exitCode=0 Jul 09 04:00:36.519829 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.519799 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"4336b78fe521beb28337182202dd2952dabefcf0c332580a97fbf47870d666df"} Jul 09 04:00:36.521174 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.521155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lm56f" event={"ID":"b87c8df8-b0b7-417f-bf10-932250ae18cc","Type":"ContainerStarted","Data":"b74b0f4054e4358e2084a32030e0feea690dd7130ea71e3674829b4db20e6192"} Jul 09 04:00:36.522783 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.522766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"e52b35c592ac5db929d2bf2d7d77f249b73799ec3eee6f071c68b1b5efe14a26"} Jul 09 04:00:36.524537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.524520 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" event={"ID":"7c8eb13dd227d9300d586f0f80eae159","Type":"ContainerStarted","Data":"1e4e270cdd3248673b5d544a83bdbdfa2bafea27c23ac2bf0a32a1ffc354ebcc"} Jul 09 04:00:36.526347 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.526329 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzh4j" event={"ID":"04568c63-abf8-4625-a487-478ab797e339","Type":"ContainerStarted","Data":"4c01ea4a27870975e8cff9ba676193fe06affa2bb4a9e890f758e949d9e47de4"} Jul 09 04:00:36.527759 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.527742 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" event={"ID":"8090bdf4-63af-4af9-949d-2453f457d4da","Type":"ContainerStarted","Data":"eeb747912e22cc61e0f17ab2e2a749ddbf50321bd8472528d26af65ced78be69"} Jul 09 04:00:36.539346 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.539300 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-drpsr" podStartSLOduration=2.116452963 podStartE2EDuration="22.539265771s" totalImagesPullingTime="20.422812808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.537620869 +0000 UTC m=+22.652797908" watchObservedRunningTime="2026-07-09 04:00:36.539265771 +0000 UTC m=+22.654442808" Jul 09 04:00:36.550561 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.550523 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-199.ec2.internal" podStartSLOduration=21.550511143 podStartE2EDuration="21.550511143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.550411106 +0000 UTC m=+22.665588142" watchObservedRunningTime="2026-07-09 04:00:36.550511143 +0000 UTC m=+22.665688178" Jul 09 04:00:36.567816 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.564891 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-199.ec2.internal" podStartSLOduration=21.564872452 podStartE2EDuration="21.564872452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.564232304 +0000 UTC m=+22.679409339" watchObservedRunningTime="2026-07-09 04:00:36.564872452 +0000 UTC m=+22.680049489" Jul 09 04:00:36.578691 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.578641 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zq56v" podStartSLOduration=2.198080761 podStartE2EDuration="22.578620492s" totalImagesPullingTime="20.380539731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.578206524 +0000 UTC m=+22.693383561" watchObservedRunningTime="2026-07-09 04:00:36.578620492 +0000 UTC m=+22.693797528" Jul 09 04:00:36.593332 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.593194 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lm56f" podStartSLOduration=2.258249616 podStartE2EDuration="22.593178719s" totalImagesPullingTime="20.334929103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.593074467 +0000 UTC m=+22.708251504" watchObservedRunningTime="2026-07-09 04:00:36.593178719 +0000 UTC m=+22.708355780" Jul 09 04:00:36.629970 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.629921 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6b6gw" podStartSLOduration=2.221822847 podStartE2EDuration="22.629907156s" totalImagesPullingTime="20.408084309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.611171276 +0000 UTC m=+22.726348312" watchObservedRunningTime="2026-07-09 04:00:36.629907156 +0000 UTC m=+22.745084192" Jul 09 04:00:36.656847 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:36.656802 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gzh4j" podStartSLOduration=10.806503101 podStartE2EDuration="22.656786008s" totalImagesPullingTime="11.850282907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:36.629716804 +0000 UTC m=+22.744893839" watchObservedRunningTime="2026-07-09 04:00:36.656786008 +0000 UTC m=+22.771963044" Jul 09 04:00:37.422082 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.422030 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:37.422282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.422055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:37.422282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.422204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:37.422390 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:37.422159 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:37.422390 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:37.422297 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:37.422390 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:37.422361 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:37.531475 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.531426 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6rkgx" event={"ID":"0e3a6cfb-3551-4d0b-8e42-7c28306cfe5a","Type":"ContainerStarted","Data":"f91def381aa00a4652e3214b5ec6523dc8d8f544b5ca446cc3caa193f2e2c1ee"} Jul 09 04:00:37.534596 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.534575 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:00:37.535099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535063 2570 generic.go:356] "Generic (PLEG): container finished" podID="4b2ff288-75bd-49e2-a498-146c506d2b02" containerID="479ba09d45ac4139b5170b52f0f63f2472e7013971ad353295a7d2812a436bd3" exitCode=1 Jul 09 04:00:37.535099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535089 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"a9624b45b304e44007b063f2fc58c8a936a1cff1a3b3f2024b4285fd21383044"} Jul 09 04:00:37.535257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"b9e0b9e59126031d0e608997b70e98876be9a9cb9b3e2bf9a0be7a994e862b99"} Jul 09 04:00:37.535257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"b6607f7138590eeabc7dbe1f9d30e6573a2d9070a17a3807cb875284c011bc88"} Jul 09 04:00:37.535359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"fa9d07ec51637772a13cdb4f0e0ba6fb847aec2e3301b4d088a5a6646255a63f"} Jul 09 04:00:37.535359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.535311 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerDied","Data":"479ba09d45ac4139b5170b52f0f63f2472e7013971ad353295a7d2812a436bd3"} Jul 09 04:00:37.835144 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:37.835125 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 04:00:38.355539 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.355394 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T04:00:37.835140416Z","UUID":"674a9dcf-81b7-4253-9ad0-6b287718d165","Handler":null,"Name":"","Endpoint":""} Jul 09 04:00:38.357347 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.357320 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 04:00:38.357482 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.357361 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 04:00:38.538913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.538881 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" event={"ID":"8090bdf4-63af-4af9-949d-2453f457d4da","Type":"ContainerStarted","Data":"90f09d1787b629f8034862511e23b5dbde1dcdaa30b43452fbdb67fbf9a5883d"} Jul 09 04:00:38.910226 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.910192 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:38.910799 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.910767 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:38.927962 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:38.927909 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6rkgx" podStartSLOduration=9.246585686 podStartE2EDuration="24.927891188s" totalImagesPullingTime="15.681305502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:37.548746319 +0000 UTC m=+23.663923354" watchObservedRunningTime="2026-07-09 04:00:38.927891188 +0000 UTC m=+25.043068223" Jul 09 04:00:39.422376 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.422175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:39.422588 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:39.422469 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:39.422588 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.422199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:39.422588 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:39.422573 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:39.422588 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.422186 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:39.422777 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:39.422641 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:39.544255 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.544224 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:00:39.544673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.544625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"dfd98803c86afc0f858777ab83ddbebacc1cbe354406e58a03ffa57f28ca8e21"} Jul 09 04:00:39.546876 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.546848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" event={"ID":"8090bdf4-63af-4af9-949d-2453f457d4da","Type":"ContainerStarted","Data":"b7e7df81ca5bdb866d14d0da7f355eca0e168819e0a5e441d863d5abb14c9e42"} Jul 09 04:00:39.547134 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.547111 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:39.547571 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.547553 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lm56f" Jul 09 04:00:39.564664 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:39.564603 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6k62" podStartSLOduration=2.727607472 podStartE2EDuration="25.564590249s" totalImagesPullingTime="22.836982777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:39.56428604 +0000 UTC m=+25.679463087" watchObservedRunningTime="2026-07-09 04:00:39.564590249 +0000 UTC m=+25.679767284" Jul 09 04:00:41.422637 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.422470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:41.423301 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.422511 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:41.423301 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:41.422722 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:41.423301 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.422513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:41.423301 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:41.422793 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:41.423301 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:41.422850 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:41.555171 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.555141 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:00:41.555566 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.555542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"dc2560dbe9aa8688a1b4cb69f1c4cb8dc9924a333dff8a8c5a39e428f17e46cb"} Jul 09 04:00:41.555942 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.555819 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:41.555942 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.555890 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:41.556069 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.556030 2570 scope.go:117] "RemoveContainer" containerID="479ba09d45ac4139b5170b52f0f63f2472e7013971ad353295a7d2812a436bd3" Jul 09 04:00:41.557528 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.557500 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="ed514c54b72d5bbb24caed7dee45765bcdb2ff56d89c1d224d958f2f6e960d3d" exitCode=0 Jul 09 04:00:41.557623 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.557583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"ed514c54b72d5bbb24caed7dee45765bcdb2ff56d89c1d224d958f2f6e960d3d"} Jul 09 04:00:41.573807 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:41.573782 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:42.566351 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:42.566321 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:00:42.566834 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:42.566662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" event={"ID":"4b2ff288-75bd-49e2-a498-146c506d2b02","Type":"ContainerStarted","Data":"ad1db08dbf96427bc2a48f9d193e9df67c307553efe8907077d92f3a5978a111"} Jul 09 04:00:42.566898 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:42.566851 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:42.582995 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:42.582968 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:00:42.592982 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:42.592932 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" podStartSLOduration=7.944157188 podStartE2EDuration="28.592912972s" totalImagesPullingTime="20.648755784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:42.591377668 +0000 UTC m=+28.706554704" watchObservedRunningTime="2026-07-09 04:00:42.592912972 +0000 UTC m=+28.708090009" Jul 09 04:00:43.112257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.112218 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r64cg"] Jul 09 04:00:43.112450 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.112357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:43.112511 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:43.112478 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:43.115605 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.115574 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z872w"] Jul 09 04:00:43.115749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.115688 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:43.115825 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:43.115797 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:43.116262 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.116242 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fccbg"] Jul 09 04:00:43.116346 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:43.116320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:43.116405 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:43.116382 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:44.423087 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:44.422903 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:44.423589 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:44.422973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:44.423589 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:44.423156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:44.423589 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:44.423221 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:44.576811 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:44.576774 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="ed0203d6c30ef946510910dda123faded412a67818b3f847b42950d5ccf6b2c8" exitCode=0 Jul 09 04:00:44.576997 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:44.576863 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"ed0203d6c30ef946510910dda123faded412a67818b3f847b42950d5ccf6b2c8"} Jul 09 04:00:45.421981 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:45.421906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:45.422140 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:45.422016 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:46.422366 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:46.422322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:46.423294 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:46.422438 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:46.423294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:46.422490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:46.423294 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:46.422579 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:46.586423 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:46.586383 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="f9c7a1cc96862b23f4f55d7c82828ce4f5ea8f75e0d3975d1ad7d5acef76023c" exitCode=0 Jul 09 04:00:46.586568 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:46.586442 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"f9c7a1cc96862b23f4f55d7c82828ce4f5ea8f75e0d3975d1ad7d5acef76023c"} Jul 09 04:00:47.133352 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:47.133314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:47.133518 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:47.133362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:47.133518 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.133455 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:47.133518 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.133490 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:47.133518 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.133520 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret podName:cb7a001d-b4c3-4aba-a69e-786b015d507a nodeName:}" failed. No retries permitted until 2026-07-09 04:01:19.133504327 +0000 UTC m=+65.248681341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret") pod "global-pull-secret-syncer-z872w" (UID: "cb7a001d-b4c3-4aba-a69e-786b015d507a") : object "kube-system"/"original-pull-secret" not registered Jul 09 04:00:47.133686 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.133544 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs podName:e3a6e221-92b6-4128-ae09-a15961f44acf nodeName:}" failed. No retries permitted until 2026-07-09 04:01:19.133527227 +0000 UTC m=+65.248704262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs") pod "network-metrics-daemon-r64cg" (UID: "e3a6e221-92b6-4128-ae09-a15961f44acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 04:00:47.238293 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:47.235019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:47.238293 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.235301 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 04:00:47.238293 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.235322 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 04:00:47.238293 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.235343 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l5gms for pod openshift-network-diagnostics/network-check-target-fccbg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:47.238293 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.235417 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms podName:83c572cc-dc49-412f-92f0-9944ad24f8a9 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:19.23538213 +0000 UTC m=+65.350559160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5gms" (UniqueName: "kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms") pod "network-check-target-fccbg" (UID: "83c572cc-dc49-412f-92f0-9944ad24f8a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 04:00:47.422539 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:47.422509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:47.423010 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:47.422636 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r64cg" podUID="e3a6e221-92b6-4128-ae09-a15961f44acf" Jul 09 04:00:48.422315 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.422281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:48.422481 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.422281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:48.422481 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.422407 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z872w" podUID="cb7a001d-b4c3-4aba-a69e-786b015d507a" Jul 09 04:00:48.422481 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.422460 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fccbg" podUID="83c572cc-dc49-412f-92f0-9944ad24f8a9" Jul 09 04:00:48.686029 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.685941 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-199.ec2.internal" event="NodeReady" Jul 09 04:00:48.686216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.686137 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 04:00:48.728339 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.728305 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fnsj2"] Jul 09 04:00:48.730190 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.730169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.733582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.733396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 04:00:48.733668 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.733452 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 04:00:48.733668 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.733530 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2b4nw\"" Jul 09 04:00:48.737514 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.737490 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qkwhv"] Jul 09 04:00:48.739622 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.739475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.740476 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.740439 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnsj2"] Jul 09 04:00:48.742465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.742225 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 04:00:48.742465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.742235 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 04:00:48.742465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.742252 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 04:00:48.742465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.742229 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwpz9\"" Jul 09 04:00:48.742465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.742234 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 04:00:48.750861 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.750844 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b8brk"] Jul 09 04:00:48.752567 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.752553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.755266 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.755235 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qkwhv"] Jul 09 04:00:48.755373 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.755298 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 04:00:48.755373 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.755322 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pc8bq\"" Jul 09 04:00:48.755542 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.755526 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 04:00:48.755655 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.755634 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 04:00:48.762714 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.762686 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b8brk"] Jul 09 04:00:48.847407 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvcx\" (UniqueName: \"kubernetes.io/projected/7075c303-5849-45fd-a889-7370137846d5-kube-api-access-tfvcx\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.847407 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2d5l\" (UniqueName: \"kubernetes.io/projected/5b7fb9b2-8775-48f1-8a57-3068090257c4-kube-api-access-l2d5l\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/469c004e-31b2-446e-b6d3-e3067cd22c44-data-volume\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czpq\" (UniqueName: \"kubernetes.io/projected/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-api-access-9czpq\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b7fb9b2-8775-48f1-8a57-3068090257c4-cert\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847615 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7075c303-5849-45fd-a889-7370137846d5-config-volume\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.847704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/469c004e-31b2-446e-b6d3-e3067cd22c44-crio-socket\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.847913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7075c303-5849-45fd-a889-7370137846d5-tmp-dir\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.847913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.847913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.847779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.948350 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvcx\" (UniqueName: \"kubernetes.io/projected/7075c303-5849-45fd-a889-7370137846d5-kube-api-access-tfvcx\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.948350 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2d5l\" (UniqueName: \"kubernetes.io/projected/5b7fb9b2-8775-48f1-8a57-3068090257c4-kube-api-access-l2d5l\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.948350 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/469c004e-31b2-446e-b6d3-e3067cd22c44-data-volume\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czpq\" (UniqueName: \"kubernetes.io/projected/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-api-access-9czpq\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b7fb9b2-8775-48f1-8a57-3068090257c4-cert\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7075c303-5849-45fd-a889-7370137846d5-config-volume\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/469c004e-31b2-446e-b6d3-e3067cd22c44-crio-socket\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7075c303-5849-45fd-a889-7370137846d5-tmp-dir\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.948636 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948602 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948650 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948675 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:49.448654431 +0000 UTC m=+35.563831461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : configmap "kube-rbac-proxy" not found Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948698 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls podName:7075c303-5849-45fd-a889-7370137846d5 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:49.448682918 +0000 UTC m=+35.563859949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls") pod "dns-default-fnsj2" (UID: "7075c303-5849-45fd-a889-7370137846d5") : secret "dns-default-metrics-tls" not found Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948777 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/469c004e-31b2-446e-b6d3-e3067cd22c44-data-volume\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.948851 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/469c004e-31b2-446e-b6d3-e3067cd22c44-crio-socket\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948894 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:48.949066 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:48.948931 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:49.448921155 +0000 UTC m=+35.564098168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:48.949374 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.949115 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7075c303-5849-45fd-a889-7370137846d5-tmp-dir\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.949374 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.949128 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7075c303-5849-45fd-a889-7370137846d5-config-volume\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.953074 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.953031 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b7fb9b2-8775-48f1-8a57-3068090257c4-cert\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.957582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.957559 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvcx\" (UniqueName: \"kubernetes.io/projected/7075c303-5849-45fd-a889-7370137846d5-kube-api-access-tfvcx\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:48.957721 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.957664 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2d5l\" (UniqueName: \"kubernetes.io/projected/5b7fb9b2-8775-48f1-8a57-3068090257c4-kube-api-access-l2d5l\") pod \"ingress-canary-b8brk\" (UID: \"5b7fb9b2-8775-48f1-8a57-3068090257c4\") " pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:48.958448 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:48.958425 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czpq\" (UniqueName: \"kubernetes.io/projected/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-api-access-9czpq\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:49.054646 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.054611 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vgc9l"] Jul 09 04:00:49.063438 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.063409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b8brk" Jul 09 04:00:49.063601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.063580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.067084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vgc9l"] Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.067417 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.067597 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8lr9q\"" Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.067780 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.067975 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 04:00:49.069822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.068173 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 04:00:49.151022 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.150995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslls\" (UniqueName: \"kubernetes.io/projected/fcb39214-d455-4cee-bb51-b909a0bc4dca-kube-api-access-rslls\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.151158 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.151080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-key\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.151158 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.151128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-cabundle\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.214702 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.214593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b8brk"] Jul 09 04:00:49.218318 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:49.218288 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7fb9b2_8775_48f1_8a57_3068090257c4.slice/crio-ec7289c0801e825b88bb9870bcd072cacb5094bb1a6250c042e21db80318a50b WatchSource:0}: Error finding container ec7289c0801e825b88bb9870bcd072cacb5094bb1a6250c042e21db80318a50b: Status 404 returned error can't find the container with id ec7289c0801e825b88bb9870bcd072cacb5094bb1a6250c042e21db80318a50b Jul 09 04:00:49.251818 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.251793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslls\" (UniqueName: \"kubernetes.io/projected/fcb39214-d455-4cee-bb51-b909a0bc4dca-kube-api-access-rslls\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.251925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.251843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-key\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.251925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.251879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-cabundle\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.252475 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.252455 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-cabundle\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.254715 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.254689 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcb39214-d455-4cee-bb51-b909a0bc4dca-signing-key\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.260195 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.260174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslls\" (UniqueName: \"kubernetes.io/projected/fcb39214-d455-4cee-bb51-b909a0bc4dca-kube-api-access-rslls\") pod \"service-ca-758f599c5-vgc9l\" (UID: \"fcb39214-d455-4cee-bb51-b909a0bc4dca\") " pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.388664 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.388637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-vgc9l" Jul 09 04:00:49.421922 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.421891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:00:49.424886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.424849 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 04:00:49.425345 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.424908 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqsq9\"" Jul 09 04:00:49.452953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.452921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:49.453119 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.452992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:49.453119 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.453026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:49.453119 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453075 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:49.453285 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453131 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:49.453285 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453143 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:50.453121579 +0000 UTC m=+36.568298614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : configmap "kube-rbac-proxy" not found Jul 09 04:00:49.453285 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453173 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:49.453285 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453179 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls podName:7075c303-5849-45fd-a889-7370137846d5 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:50.453168388 +0000 UTC m=+36.568345403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls") pod "dns-default-fnsj2" (UID: "7075c303-5849-45fd-a889-7370137846d5") : secret "dns-default-metrics-tls" not found Jul 09 04:00:49.453285 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:49.453239 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:50.453226228 +0000 UTC m=+36.568403246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:49.596631 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:49.596551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b8brk" event={"ID":"5b7fb9b2-8775-48f1-8a57-3068090257c4","Type":"ContainerStarted","Data":"ec7289c0801e825b88bb9870bcd072cacb5094bb1a6250c042e21db80318a50b"} Jul 09 04:00:50.422262 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.422230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:00:50.422462 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.422439 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:00:50.427226 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.427202 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 04:00:50.427226 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.427224 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 04:00:50.427700 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.427257 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vbqhw\"" Jul 09 04:00:50.427700 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.427202 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 04:00:50.463284 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.463243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:50.463466 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.463337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:50.463466 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.463375 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:50.463466 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463385 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 04:00:50.463466 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463459 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:50.463658 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463471 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:52.463450599 +0000 UTC m=+38.578627615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : configmap "kube-rbac-proxy" not found Jul 09 04:00:50.463658 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463487 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:50.463658 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463518 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls podName:7075c303-5849-45fd-a889-7370137846d5 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:52.463499417 +0000 UTC m=+38.578676431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls") pod "dns-default-fnsj2" (UID: "7075c303-5849-45fd-a889-7370137846d5") : secret "dns-default-metrics-tls" not found Jul 09 04:00:50.463658 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:50.463536 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:52.463525705 +0000 UTC m=+38.578702738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:50.878239 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:50.878065 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vgc9l"] Jul 09 04:00:50.883530 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:00:50.883500 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb39214_d455_4cee_bb51_b909a0bc4dca.slice/crio-3b6b485a094c7f87a076b2e67fe9dc66afad28ff5f07bbe22dd32f2f15de615c WatchSource:0}: Error finding container 3b6b485a094c7f87a076b2e67fe9dc66afad28ff5f07bbe22dd32f2f15de615c: Status 404 returned error can't find the container with id 3b6b485a094c7f87a076b2e67fe9dc66afad28ff5f07bbe22dd32f2f15de615c Jul 09 04:00:51.222910 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:51.222879 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zq56v_bcb69976-0b5d-41c7-97de-84e379767d58/dns-node-resolver/0.log" Jul 09 04:00:51.604246 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:51.604173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-vgc9l" event={"ID":"fcb39214-d455-4cee-bb51-b909a0bc4dca","Type":"ContainerStarted","Data":"3b6b485a094c7f87a076b2e67fe9dc66afad28ff5f07bbe22dd32f2f15de615c"} Jul 09 04:00:51.609117 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:51.609079 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="54c348ce9951e831478ade39a7b32ee84311b409e2cff8495ca51757ead91263" exitCode=0 Jul 09 04:00:51.609246 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:51.609147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"54c348ce9951e831478ade39a7b32ee84311b409e2cff8495ca51757ead91263"} Jul 09 04:00:52.203722 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.203691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzh4j_04568c63-abf8-4625-a487-478ab797e339/node-ca/0.log" Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.480305 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.480386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.480460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:52.480466 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:52.480533 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:56.480514556 +0000 UTC m=+42.595691594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:52.480540 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:52.480570 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls podName:7075c303-5849-45fd-a889-7370137846d5 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:56.480560546 +0000 UTC m=+42.595737560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls") pod "dns-default-fnsj2" (UID: "7075c303-5849-45fd-a889-7370137846d5") : secret "dns-default-metrics-tls" not found Jul 09 04:00:52.481179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.481088 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/469c004e-31b2-446e-b6d3-e3067cd22c44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:52.615377 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.615339 2570 generic.go:356] "Generic (PLEG): container finished" podID="eeba12f8-f6d5-413f-b135-50720a67ab29" containerID="90418ee2f59984c1a88390cdbbc307274f6d5b81effd7e3d9199018427972dba" exitCode=0 Jul 09 04:00:52.615989 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:52.615407 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerDied","Data":"90418ee2f59984c1a88390cdbbc307274f6d5b81effd7e3d9199018427972dba"} Jul 09 04:00:53.620430 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.620396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b8brk" event={"ID":"5b7fb9b2-8775-48f1-8a57-3068090257c4","Type":"ContainerStarted","Data":"3e7d0826021a5d1fd7fa3cb41a957a5ad3d0186952f103be01c4ec7395dcb6be"} Jul 09 04:00:53.622659 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.622635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-vgc9l" event={"ID":"fcb39214-d455-4cee-bb51-b909a0bc4dca","Type":"ContainerStarted","Data":"f673846d13370110b9d4c9763fe126491595a4eaf51b7b66801ebfd6798a595b"} Jul 09 04:00:53.626428 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.626407 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82x8f" event={"ID":"eeba12f8-f6d5-413f-b135-50720a67ab29","Type":"ContainerStarted","Data":"646b459026813c2e5dfd8e9c8bd03cf7941586e9c151992608d916918c2746f1"} Jul 09 04:00:53.638724 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.638667 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b8brk" podStartSLOduration=1.376776894 podStartE2EDuration="5.638646478s" totalImagesPullingTime="4.261869584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:53.637297789 +0000 UTC m=+39.752474826" watchObservedRunningTime="2026-07-09 04:00:53.638646478 +0000 UTC m=+39.753823515" Jul 09 04:00:53.667616 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.667561 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-82x8f" podStartSLOduration=12.46044459 podStartE2EDuration="39.667540549s" totalImagesPullingTime="27.207095959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:53.667285707 +0000 UTC m=+39.782462757" watchObservedRunningTime="2026-07-09 04:00:53.667540549 +0000 UTC m=+39.782717586" Jul 09 04:00:53.695720 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:53.695667 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-vgc9l" podStartSLOduration=2.096128393 podStartE2EDuration="4.695652654s" totalImagesPullingTime="2.599524261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:53.693903855 +0000 UTC m=+39.809080892" watchObservedRunningTime="2026-07-09 04:00:53.695652654 +0000 UTC m=+39.810829739" Jul 09 04:00:56.515548 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:56.515508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:00:56.515548 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:00:56.515556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:00:56.516021 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:56.515659 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 04:00:56.516021 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:56.515675 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 04:00:56.516021 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:56.515712 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls podName:469c004e-31b2-446e-b6d3-e3067cd22c44 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:04.515697978 +0000 UTC m=+50.630874991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qkwhv" (UID: "469c004e-31b2-446e-b6d3-e3067cd22c44") : secret "insights-runtime-extractor-tls" not found Jul 09 04:00:56.516021 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:00:56.515740 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls podName:7075c303-5849-45fd-a889-7370137846d5 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:04.515721649 +0000 UTC m=+50.630898663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls") pod "dns-default-fnsj2" (UID: "7075c303-5849-45fd-a889-7370137846d5") : secret "dns-default-metrics-tls" not found Jul 09 04:01:04.581920 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.581878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:01:04.582408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.581976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:01:04.585193 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.585166 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7075c303-5849-45fd-a889-7370137846d5-metrics-tls\") pod \"dns-default-fnsj2\" (UID: \"7075c303-5849-45fd-a889-7370137846d5\") " pod="openshift-dns/dns-default-fnsj2" Jul 09 04:01:04.585292 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.585214 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/469c004e-31b2-446e-b6d3-e3067cd22c44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qkwhv\" (UID: \"469c004e-31b2-446e-b6d3-e3067cd22c44\") " pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:01:04.641267 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.641233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnsj2" Jul 09 04:01:04.649394 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.649367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qkwhv" Jul 09 04:01:04.822628 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.822600 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnsj2"] Jul 09 04:01:04.826808 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:04.826778 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7075c303_5849_45fd_a889_7370137846d5.slice/crio-475cbd32eabf151c03323b9206a35c4ef0bcb77c3b8d6c845abbe7a5b9258779 WatchSource:0}: Error finding container 475cbd32eabf151c03323b9206a35c4ef0bcb77c3b8d6c845abbe7a5b9258779: Status 404 returned error can't find the container with id 475cbd32eabf151c03323b9206a35c4ef0bcb77c3b8d6c845abbe7a5b9258779 Jul 09 04:01:04.836603 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:04.836573 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qkwhv"] Jul 09 04:01:04.839303 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:04.839276 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469c004e_31b2_446e_b6d3_e3067cd22c44.slice/crio-9e3203e2015eff972ff48dceddc20f1b08b11b096d156beda9b568d390151c12 WatchSource:0}: Error finding container 9e3203e2015eff972ff48dceddc20f1b08b11b096d156beda9b568d390151c12: Status 404 returned error can't find the container with id 9e3203e2015eff972ff48dceddc20f1b08b11b096d156beda9b568d390151c12 Jul 09 04:01:05.659278 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:05.659240 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnsj2" event={"ID":"7075c303-5849-45fd-a889-7370137846d5","Type":"ContainerStarted","Data":"475cbd32eabf151c03323b9206a35c4ef0bcb77c3b8d6c845abbe7a5b9258779"} Jul 09 04:01:05.660636 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:05.660606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qkwhv" event={"ID":"469c004e-31b2-446e-b6d3-e3067cd22c44","Type":"ContainerStarted","Data":"9e3203e2015eff972ff48dceddc20f1b08b11b096d156beda9b568d390151c12"} Jul 09 04:01:09.675563 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:09.675505 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnsj2" event={"ID":"7075c303-5849-45fd-a889-7370137846d5","Type":"ContainerStarted","Data":"4f3106f9fa2ae8c8edf8016c42b223334a68f37263a72f5642404c5315703407"} Jul 09 04:01:09.677990 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:09.677960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qkwhv" event={"ID":"469c004e-31b2-446e-b6d3-e3067cd22c44","Type":"ContainerStarted","Data":"d65d3319dfc3d06bc6e2598bf1a7a1d53137c0f75527af2d1220cdd5369071b6"} Jul 09 04:01:10.192998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.192964 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn"] Jul 09 04:01:10.195241 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.195217 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.195705 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.195679 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh"] Jul 09 04:01:10.197326 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.197306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.198408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.198241 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 04:01:10.198408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.198305 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 04:01:10.198408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.198316 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-h2kk2\"" Jul 09 04:01:10.198408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.198320 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 04:01:10.198408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.198357 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 04:01:10.199819 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.199722 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 04:01:10.207519 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.207501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn"] Jul 09 04:01:10.210089 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.209970 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh"] Jul 09 04:01:10.227085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.227038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hzx\" (UniqueName: \"kubernetes.io/projected/56c1bc10-c9e9-4df4-88df-34c106601f0b-kube-api-access-d5hzx\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.227225 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.227111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fefb8de1-541a-4b87-b903-ed2ea69082df-tmp\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.227225 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.227155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c1bc10-c9e9-4df4-88df-34c106601f0b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.227340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.227226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fefb8de1-541a-4b87-b903-ed2ea69082df-klusterlet-config\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.227340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.227261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l2d\" (UniqueName: \"kubernetes.io/projected/fefb8de1-541a-4b87-b903-ed2ea69082df-kube-api-access-p8l2d\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.298766 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.298737 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64ddc67c5d-2vs8n"] Jul 09 04:01:10.300658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.300641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.303238 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.303215 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 04:01:10.303452 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.303436 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k8j4f\"" Jul 09 04:01:10.303504 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.303469 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 04:01:10.303868 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.303854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 04:01:10.312035 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.312015 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 04:01:10.315266 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.315247 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64ddc67c5d-2vs8n"] Jul 09 04:01:10.328607 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fefb8de1-541a-4b87-b903-ed2ea69082df-klusterlet-config\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.328747 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fefb8de1-541a-4b87-b903-ed2ea69082df-tmp\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.328747 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328665 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-registry-tls\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328747 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-registry-certificates\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328747 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xtj\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-kube-api-access-x6xtj\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l2d\" (UniqueName: \"kubernetes.io/projected/fefb8de1-541a-4b87-b903-ed2ea69082df-kube-api-access-p8l2d\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-image-registry-private-configuration\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-trusted-ca\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hzx\" (UniqueName: \"kubernetes.io/projected/56c1bc10-c9e9-4df4-88df-34c106601f0b-kube-api-access-d5hzx\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9212353-2fa5-4c96-a873-995fef5359ce-ca-trust-extracted\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.328953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-installation-pull-secrets\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.329207 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.328964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-bound-sa-token\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.329207 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.329007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c1bc10-c9e9-4df4-88df-34c106601f0b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.329207 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.329031 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fefb8de1-541a-4b87-b903-ed2ea69082df-tmp\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.331343 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.331324 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fefb8de1-541a-4b87-b903-ed2ea69082df-klusterlet-config\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.331462 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.331444 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56c1bc10-c9e9-4df4-88df-34c106601f0b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.338453 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.338431 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l2d\" (UniqueName: \"kubernetes.io/projected/fefb8de1-541a-4b87-b903-ed2ea69082df-kube-api-access-p8l2d\") pod \"klusterlet-addon-workmgr-7476d96f67-vqqqh\" (UID: \"fefb8de1-541a-4b87-b903-ed2ea69082df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.338544 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.338530 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hzx\" (UniqueName: \"kubernetes.io/projected/56c1bc10-c9e9-4df4-88df-34c106601f0b-kube-api-access-d5hzx\") pod \"managed-serviceaccount-addon-agent-588f46f89b-tb6bn\" (UID: \"56c1bc10-c9e9-4df4-88df-34c106601f0b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.430165 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-registry-tls\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-registry-certificates\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xtj\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-kube-api-access-x6xtj\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-image-registry-private-configuration\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-trusted-ca\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9212353-2fa5-4c96-a873-995fef5359ce-ca-trust-extracted\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-installation-pull-secrets\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-bound-sa-token\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.430887 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.430852 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9212353-2fa5-4c96-a873-995fef5359ce-ca-trust-extracted\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.431317 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.431247 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-registry-certificates\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.431430 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.431361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9212353-2fa5-4c96-a873-995fef5359ce-trusted-ca\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.432792 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.432767 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-registry-tls\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.432903 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.432886 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-image-registry-private-configuration\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.432976 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.432958 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9212353-2fa5-4c96-a873-995fef5359ce-installation-pull-secrets\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.450625 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.450568 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xtj\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-kube-api-access-x6xtj\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.452830 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.452814 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9212353-2fa5-4c96-a873-995fef5359ce-bound-sa-token\") pod \"image-registry-64ddc67c5d-2vs8n\" (UID: \"b9212353-2fa5-4c96-a873-995fef5359ce\") " pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.527767 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.527733 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" Jul 09 04:01:10.532919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.532896 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:10.611463 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.611432 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:10.676499 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.676467 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh"] Jul 09 04:01:10.681482 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:10.681443 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefb8de1_541a_4b87_b903_ed2ea69082df.slice/crio-b6e33cff48ca4f39e9832a7875b302c08ca8cb2144f5ae9defc5ae0bee518520 WatchSource:0}: Error finding container b6e33cff48ca4f39e9832a7875b302c08ca8cb2144f5ae9defc5ae0bee518520: Status 404 returned error can't find the container with id b6e33cff48ca4f39e9832a7875b302c08ca8cb2144f5ae9defc5ae0bee518520 Jul 09 04:01:10.685138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.685067 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnsj2" event={"ID":"7075c303-5849-45fd-a889-7370137846d5","Type":"ContainerStarted","Data":"bae27ffb6e739ce512c378fcfc5d49e4532b95dfc6f72760dd9a4431eda43de0"} Jul 09 04:01:10.685250 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.685220 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fnsj2" Jul 09 04:01:10.697142 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.697112 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn"] Jul 09 04:01:10.700887 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:10.700822 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56c1bc10_c9e9_4df4_88df_34c106601f0b.slice/crio-be7366c00b525fe7d52d9aaea15b76981b71dbdde329d6e602afa33146b86564 WatchSource:0}: Error finding container be7366c00b525fe7d52d9aaea15b76981b71dbdde329d6e602afa33146b86564: Status 404 returned error can't find the container with id be7366c00b525fe7d52d9aaea15b76981b71dbdde329d6e602afa33146b86564 Jul 09 04:01:10.702871 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.702734 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fnsj2" podStartSLOduration=17.983659241 podStartE2EDuration="22.702713515s" totalImagesPullingTime="4.719054274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:10.701820219 +0000 UTC m=+56.816997281" watchObservedRunningTime="2026-07-09 04:01:10.702713515 +0000 UTC m=+56.817890551" Jul 09 04:01:10.749138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:10.749099 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64ddc67c5d-2vs8n"] Jul 09 04:01:10.752622 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:10.752594 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9212353_2fa5_4c96_a873_995fef5359ce.slice/crio-8d872aaa382448c68b69f80085e0a3f8c9339f856bf4822d308585b06520cc71 WatchSource:0}: Error finding container 8d872aaa382448c68b69f80085e0a3f8c9339f856bf4822d308585b06520cc71: Status 404 returned error can't find the container with id 8d872aaa382448c68b69f80085e0a3f8c9339f856bf4822d308585b06520cc71 Jul 09 04:01:11.691134 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.691097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" event={"ID":"fefb8de1-541a-4b87-b903-ed2ea69082df","Type":"ContainerStarted","Data":"b6e33cff48ca4f39e9832a7875b302c08ca8cb2144f5ae9defc5ae0bee518520"} Jul 09 04:01:11.692571 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.692536 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" event={"ID":"56c1bc10-c9e9-4df4-88df-34c106601f0b","Type":"ContainerStarted","Data":"be7366c00b525fe7d52d9aaea15b76981b71dbdde329d6e602afa33146b86564"} Jul 09 04:01:11.694649 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.694619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" event={"ID":"b9212353-2fa5-4c96-a873-995fef5359ce","Type":"ContainerStarted","Data":"2203b88bc1eed4999557593bf7015f2742dc24724e976ccbae2787e280229b64"} Jul 09 04:01:11.694749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.694687 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" event={"ID":"b9212353-2fa5-4c96-a873-995fef5359ce","Type":"ContainerStarted","Data":"8d872aaa382448c68b69f80085e0a3f8c9339f856bf4822d308585b06520cc71"} Jul 09 04:01:11.695586 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.695553 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:11.701905 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.701842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qkwhv" event={"ID":"469c004e-31b2-446e-b6d3-e3067cd22c44","Type":"ContainerStarted","Data":"6ac722b397c480337b2c7b42d35eb0c9fafb51908ccc02b9d331ded7a5c44f31"} Jul 09 04:01:11.717548 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:11.716635 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" podStartSLOduration=1.716614183 podStartE2EDuration="1.716614183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:11.716139325 +0000 UTC m=+57.831316365" watchObservedRunningTime="2026-07-09 04:01:11.716614183 +0000 UTC m=+57.831791220" Jul 09 04:01:14.598226 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:14.598188 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr2n4" Jul 09 04:01:15.718776 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:15.718737 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qkwhv" event={"ID":"469c004e-31b2-446e-b6d3-e3067cd22c44","Type":"ContainerStarted","Data":"979f46335089e1d33e69057798372a2635dfd2c73e04107fb388c1ca49924839"} Jul 09 04:01:15.720159 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:15.720130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" event={"ID":"56c1bc10-c9e9-4df4-88df-34c106601f0b","Type":"ContainerStarted","Data":"a0d97d6780dc31777f96560ff5981018ff04f6a83656c4e6a99bbeabc8133475"} Jul 09 04:01:15.735948 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:15.735900 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qkwhv" podStartSLOduration=17.604279774 podStartE2EDuration="27.735878834s" totalImagesPullingTime="10.13159906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:15.735009153 +0000 UTC m=+61.850186202" watchObservedRunningTime="2026-07-09 04:01:15.735878834 +0000 UTC m=+61.851055871" Jul 09 04:01:15.749123 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:15.749079 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-588f46f89b-tb6bn" podStartSLOduration=1.348274129 podStartE2EDuration="5.74906544s" totalImagesPullingTime="4.400791311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:15.74826399 +0000 UTC m=+61.863441025" watchObservedRunningTime="2026-07-09 04:01:15.74906544 +0000 UTC m=+61.864242470" Jul 09 04:01:16.717913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.717874 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-f74hh"] Jul 09 04:01:16.721085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.721058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:16.723674 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.723642 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 04:01:16.723793 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.723775 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 04:01:16.724029 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.724009 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7qv24\"" Jul 09 04:01:16.729451 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.729343 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-f74hh"] Jul 09 04:01:16.790463 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.790425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ft5j\" (UniqueName: \"kubernetes.io/projected/bcd61866-271e-4b98-9092-63140cf94d74-kube-api-access-9ft5j\") pod \"downloads-685bfb5f64-f74hh\" (UID: \"bcd61866-271e-4b98-9092-63140cf94d74\") " pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:16.891829 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.891788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ft5j\" (UniqueName: \"kubernetes.io/projected/bcd61866-271e-4b98-9092-63140cf94d74-kube-api-access-9ft5j\") pod \"downloads-685bfb5f64-f74hh\" (UID: \"bcd61866-271e-4b98-9092-63140cf94d74\") " pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:16.900746 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:16.900717 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ft5j\" (UniqueName: \"kubernetes.io/projected/bcd61866-271e-4b98-9092-63140cf94d74-kube-api-access-9ft5j\") pod \"downloads-685bfb5f64-f74hh\" (UID: \"bcd61866-271e-4b98-9092-63140cf94d74\") " pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:17.032796 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.032712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:17.214653 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.214622 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-f74hh"] Jul 09 04:01:17.219288 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:17.219264 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd61866_271e_4b98_9092_63140cf94d74.slice/crio-e52b41d5cabbdef110d3d064a413231c02dd99780812c859ffd10a51d1fb948b WatchSource:0}: Error finding container e52b41d5cabbdef110d3d064a413231c02dd99780812c859ffd10a51d1fb948b: Status 404 returned error can't find the container with id e52b41d5cabbdef110d3d064a413231c02dd99780812c859ffd10a51d1fb948b Jul 09 04:01:17.728474 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.728436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-f74hh" event={"ID":"bcd61866-271e-4b98-9092-63140cf94d74","Type":"ContainerStarted","Data":"e52b41d5cabbdef110d3d064a413231c02dd99780812c859ffd10a51d1fb948b"} Jul 09 04:01:17.729783 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.729761 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" event={"ID":"fefb8de1-541a-4b87-b903-ed2ea69082df","Type":"ContainerStarted","Data":"c4e749cc4f792412fc37e04ba869a99f8575aabd92193d1f52b8a616411b5db0"} Jul 09 04:01:17.730099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.730067 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:17.731520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.731502 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" Jul 09 04:01:17.747002 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:17.746959 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7476d96f67-vqqqh" podStartSLOduration=1.300955927 podStartE2EDuration="7.746941616s" totalImagesPullingTime="6.445985689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:17.745847596 +0000 UTC m=+63.861024634" watchObservedRunningTime="2026-07-09 04:01:17.746941616 +0000 UTC m=+63.862118655" Jul 09 04:01:19.212689 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.212650 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:01:19.212689 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.212690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:01:19.215421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.215388 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 04:01:19.215535 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.215461 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 04:01:19.226263 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.226206 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a6e221-92b6-4128-ae09-a15961f44acf-metrics-certs\") pod \"network-metrics-daemon-r64cg\" (UID: \"e3a6e221-92b6-4128-ae09-a15961f44acf\") " pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:01:19.226263 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.226233 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb7a001d-b4c3-4aba-a69e-786b015d507a-original-pull-secret\") pod \"global-pull-secret-syncer-z872w\" (UID: \"cb7a001d-b4c3-4aba-a69e-786b015d507a\") " pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:01:19.238223 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.238195 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z872w" Jul 09 04:01:19.313130 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.313100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:01:19.316176 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.316151 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 04:01:19.327018 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.326995 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 04:01:19.337317 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.337260 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gms\" (UniqueName: \"kubernetes.io/projected/83c572cc-dc49-412f-92f0-9944ad24f8a9-kube-api-access-l5gms\") pod \"network-check-target-fccbg\" (UID: \"83c572cc-dc49-412f-92f0-9944ad24f8a9\") " pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:01:19.383132 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.383096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z872w"] Jul 09 04:01:19.386356 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:19.386326 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7a001d_b4c3_4aba_a69e_786b015d507a.slice/crio-2724a54c9e452e36ef39e3b036a23c8ab61fb84cbd002477ea8cc4e37bdae2f8 WatchSource:0}: Error finding container 2724a54c9e452e36ef39e3b036a23c8ab61fb84cbd002477ea8cc4e37bdae2f8: Status 404 returned error can't find the container with id 2724a54c9e452e36ef39e3b036a23c8ab61fb84cbd002477ea8cc4e37bdae2f8 Jul 09 04:01:19.435947 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.435919 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqsq9\"" Jul 09 04:01:19.444116 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.444091 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r64cg" Jul 09 04:01:19.535928 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.535904 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vbqhw\"" Jul 09 04:01:19.544029 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.543994 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:01:19.579718 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.579634 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r64cg"] Jul 09 04:01:19.582774 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:19.582741 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a6e221_92b6_4128_ae09_a15961f44acf.slice/crio-ed8202babffb6cc48b427b4934dda0b7ec0ed0add67e7e4c4d46a17e7910cd64 WatchSource:0}: Error finding container ed8202babffb6cc48b427b4934dda0b7ec0ed0add67e7e4c4d46a17e7910cd64: Status 404 returned error can't find the container with id ed8202babffb6cc48b427b4934dda0b7ec0ed0add67e7e4c4d46a17e7910cd64 Jul 09 04:01:19.689736 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.689705 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fccbg"] Jul 09 04:01:19.692750 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:19.692720 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c572cc_dc49_412f_92f0_9944ad24f8a9.slice/crio-2b95a10982ebb724480b0c7268781a613446952bd477ac4308ceb8e71b57a036 WatchSource:0}: Error finding container 2b95a10982ebb724480b0c7268781a613446952bd477ac4308ceb8e71b57a036: Status 404 returned error can't find the container with id 2b95a10982ebb724480b0c7268781a613446952bd477ac4308ceb8e71b57a036 Jul 09 04:01:19.737434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.737402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r64cg" event={"ID":"e3a6e221-92b6-4128-ae09-a15961f44acf","Type":"ContainerStarted","Data":"ed8202babffb6cc48b427b4934dda0b7ec0ed0add67e7e4c4d46a17e7910cd64"} Jul 09 04:01:19.738494 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.738470 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z872w" event={"ID":"cb7a001d-b4c3-4aba-a69e-786b015d507a","Type":"ContainerStarted","Data":"2724a54c9e452e36ef39e3b036a23c8ab61fb84cbd002477ea8cc4e37bdae2f8"} Jul 09 04:01:19.739479 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:19.739459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fccbg" event={"ID":"83c572cc-dc49-412f-92f0-9944ad24f8a9","Type":"ContainerStarted","Data":"2b95a10982ebb724480b0c7268781a613446952bd477ac4308ceb8e71b57a036"} Jul 09 04:01:20.704461 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:20.704423 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fnsj2" Jul 09 04:01:21.668466 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.668431 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w"] Jul 09 04:01:21.686673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.685228 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w"] Jul 09 04:01:21.686673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.685355 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.689551 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.688955 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 04:01:21.689551 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.689379 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2tf64\"" Jul 09 04:01:21.691994 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.690560 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 04:01:21.691994 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.690753 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 04:01:21.691994 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.690957 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 04:01:21.691994 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.691175 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 04:01:21.754754 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.753918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r64cg" event={"ID":"e3a6e221-92b6-4128-ae09-a15961f44acf","Type":"ContainerStarted","Data":"4f18c78ad820ef8daa41586485c30ddae9d58976e0318c9d861dadbadb346406"} Jul 09 04:01:21.754754 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.753985 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r64cg" event={"ID":"e3a6e221-92b6-4128-ae09-a15961f44acf","Type":"ContainerStarted","Data":"7c6495e7ea73c9f2825d87535d801fee29a99dc1f03de0148e16ab60602c9241"} Jul 09 04:01:21.834072 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.833227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.834072 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.833294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhl2\" (UniqueName: \"kubernetes.io/projected/1322387e-ffd8-421c-b6f1-1e387194680c-kube-api-access-hnhl2\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.834072 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.833332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1322387e-ffd8-421c-b6f1-1e387194680c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.834072 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.833443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.934417 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.934325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.934417 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.934387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.934417 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.934414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhl2\" (UniqueName: \"kubernetes.io/projected/1322387e-ffd8-421c-b6f1-1e387194680c-kube-api-access-hnhl2\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.934689 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.934444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1322387e-ffd8-421c-b6f1-1e387194680c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.934877 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:21.934850 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 04:01:21.934953 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:21.934940 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls podName:1322387e-ffd8-421c-b6f1-1e387194680c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:22.434919915 +0000 UTC m=+68.550096929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-rhl9w" (UID: "1322387e-ffd8-421c-b6f1-1e387194680c") : secret "prometheus-operator-tls" not found Jul 09 04:01:21.935281 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.935183 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1322387e-ffd8-421c-b6f1-1e387194680c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.937767 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.937719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:21.948247 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:21.948199 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhl2\" (UniqueName: \"kubernetes.io/projected/1322387e-ffd8-421c-b6f1-1e387194680c-kube-api-access-hnhl2\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:22.438943 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:22.438825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:22.442427 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:22.442396 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1322387e-ffd8-421c-b6f1-1e387194680c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rhl9w\" (UID: \"1322387e-ffd8-421c-b6f1-1e387194680c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:22.602582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:22.602325 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" Jul 09 04:01:25.387653 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.386829 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r64cg" podStartSLOduration=69.922282178 podStartE2EDuration="1m11.386809261s" totalImagesPullingTime="1.464527083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:21.771558687 +0000 UTC m=+67.886735724" watchObservedRunningTime="2026-07-09 04:01:25.386809261 +0000 UTC m=+71.501986299" Jul 09 04:01:25.387653 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.387606 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w"] Jul 09 04:01:25.391101 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:25.391074 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1322387e_ffd8_421c_b6f1_1e387194680c.slice/crio-13efd862e368c9c58b2c43dd4474ed7159da03a8c579c71e94379f28602dbe4a WatchSource:0}: Error finding container 13efd862e368c9c58b2c43dd4474ed7159da03a8c579c71e94379f28602dbe4a: Status 404 returned error can't find the container with id 13efd862e368c9c58b2c43dd4474ed7159da03a8c579c71e94379f28602dbe4a Jul 09 04:01:25.781654 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.781618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z872w" event={"ID":"cb7a001d-b4c3-4aba-a69e-786b015d507a","Type":"ContainerStarted","Data":"800cc5a4192be62e24bf0ae669214bb214970be7296317dca5825a2bac5478d4"} Jul 09 04:01:25.782843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.782817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" event={"ID":"1322387e-ffd8-421c-b6f1-1e387194680c","Type":"ContainerStarted","Data":"13efd862e368c9c58b2c43dd4474ed7159da03a8c579c71e94379f28602dbe4a"} Jul 09 04:01:25.784270 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.784250 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fccbg" event={"ID":"83c572cc-dc49-412f-92f0-9944ad24f8a9","Type":"ContainerStarted","Data":"f06614e0539b236ebb61133c4fc75882372dd07838dbb6c019b2df4738c0a295"} Jul 09 04:01:25.784391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.784378 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:01:25.795910 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.795779 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z872w" podStartSLOduration=65.939879438 podStartE2EDuration="1m11.795763751s" totalImagesPullingTime="5.855884313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:25.795684347 +0000 UTC m=+71.910861384" watchObservedRunningTime="2026-07-09 04:01:25.795763751 +0000 UTC m=+71.910940791" Jul 09 04:01:25.814068 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:25.814008 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fccbg" podStartSLOduration=66.266699273 podStartE2EDuration="1m11.813995938s" totalImagesPullingTime="5.547296665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:25.813294825 +0000 UTC m=+71.928471863" watchObservedRunningTime="2026-07-09 04:01:25.813995938 +0000 UTC m=+71.929172974" Jul 09 04:01:26.845745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.845707 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:01:26.848086 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.848061 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.851520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.851341 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 04:01:26.852873 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.852737 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd6vl\"" Jul 09 04:01:26.852873 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.852774 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 04:01:26.852873 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.852810 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 04:01:26.853140 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.852884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 04:01:26.853140 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.852952 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 04:01:26.858142 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.858122 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 04:01:26.866360 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.866338 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:01:26.974936 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.974913 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.974949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.974971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.975026 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.975061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975271 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.975087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqp9\" (UniqueName: \"kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:26.975271 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:26.975124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.075638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.075598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.075803 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.075676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.075977 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.075950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.076080 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.076022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.076454 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.076396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.078285 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.076626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.078285 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.076668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqp9\" (UniqueName: \"kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.078285 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.077748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.078285 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.077941 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.078531 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.078388 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.079207 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.079091 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.082562 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.081455 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.084625 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.084586 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.091202 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.091174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqp9\" (UniqueName: \"kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9\") pod \"console-6d45678959-2d282\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.161731 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.161694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:27.315580 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.315551 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:01:27.318202 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:27.318176 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc360507b_849f_4291_9b3c_11e77702d388.slice/crio-0d7b79183b914d2aef1fc99ed95852f6c042f40f0523d93a55109df84ac827a0 WatchSource:0}: Error finding container 0d7b79183b914d2aef1fc99ed95852f6c042f40f0523d93a55109df84ac827a0: Status 404 returned error can't find the container with id 0d7b79183b914d2aef1fc99ed95852f6c042f40f0523d93a55109df84ac827a0 Jul 09 04:01:27.793215 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.793157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45678959-2d282" event={"ID":"c360507b-849f-4291-9b3c-11e77702d388","Type":"ContainerStarted","Data":"0d7b79183b914d2aef1fc99ed95852f6c042f40f0523d93a55109df84ac827a0"} Jul 09 04:01:27.795391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.795301 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" event={"ID":"1322387e-ffd8-421c-b6f1-1e387194680c","Type":"ContainerStarted","Data":"2b5bd2baa17a0fd528cefcccb1323f99f3bb305f569b26cf6bb08b15bccf277c"} Jul 09 04:01:27.795391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.795356 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" event={"ID":"1322387e-ffd8-421c-b6f1-1e387194680c","Type":"ContainerStarted","Data":"238f193b67c6a8779308cd3510ed10ce66fdf79de5a50505640499f8c9ddd2b2"} Jul 09 04:01:27.818065 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:27.817997 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rhl9w" podStartSLOduration=5.278992976 podStartE2EDuration="6.817981098s" totalImagesPullingTime="1.538988122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:27.816034747 +0000 UTC m=+73.931211782" watchObservedRunningTime="2026-07-09 04:01:27.817981098 +0000 UTC m=+73.933158134" Jul 09 04:01:30.133257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.133227 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n"] Jul 09 04:01:30.135923 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.135901 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.138550 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.138528 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 04:01:30.138683 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.138665 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 04:01:30.138798 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.138783 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ngx2d\"" Jul 09 04:01:30.140218 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.140198 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 04:01:30.148329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.147097 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h94cr"] Jul 09 04:01:30.149464 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.149444 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n"] Jul 09 04:01:30.149572 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.149561 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.152411 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.152321 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 04:01:30.153004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.152658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 04:01:30.153004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.152743 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wpfxj\"" Jul 09 04:01:30.153178 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.153017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2c08bf2-554e-4749-8e40-084903a926e0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204320 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.204523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.204422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2k2\" (UniqueName: \"kubernetes.io/projected/b2c08bf2-554e-4749-8e40-084903a926e0-kube-api-access-vs2k2\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-accelerators-collector-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2c08bf2-554e-4749-8e40-084903a926e0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-sys\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-textfile\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-root\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2k2\" (UniqueName: \"kubernetes.io/projected/b2c08bf2-554e-4749-8e40-084903a926e0-kube-api-access-vs2k2\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-metrics-client-ca\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305680 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpf2q\" (UniqueName: \"kubernetes.io/projected/4fab6402-db93-4b93-be4f-41eaaff1265e-kube-api-access-qpf2q\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307062 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-wtmp\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.307880 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.305752 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2c08bf2-554e-4749-8e40-084903a926e0-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307880 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:30.305891 2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 04:01:30.307880 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:30.305952 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls podName:b2c08bf2-554e-4749-8e40-084903a926e0 nodeName:}" failed. No retries permitted until 2026-07-09 04:01:30.805928635 +0000 UTC m=+76.921105654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-mvt5n" (UID: "b2c08bf2-554e-4749-8e40-084903a926e0") : secret "kube-state-metrics-tls" not found Jul 09 04:01:30.307880 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.306828 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.307880 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.306995 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.314126 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.314076 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.318431 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.318351 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2k2\" (UniqueName: \"kubernetes.io/projected/b2c08bf2-554e-4749-8e40-084903a926e0-kube-api-access-vs2k2\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-textfile\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-root\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-metrics-client-ca\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpf2q\" (UniqueName: \"kubernetes.io/projected/4fab6402-db93-4b93-be4f-41eaaff1265e-kube-api-access-qpf2q\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408671 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-wtmp\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408699 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-accelerators-collector-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408744 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-sys\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.408824 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-sys\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.409002 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-root\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:30.409128 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:01:30.409185 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls podName:4fab6402-db93-4b93-be4f-41eaaff1265e nodeName:}" failed. No retries permitted until 2026-07-09 04:01:30.909166416 +0000 UTC m=+77.024343435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls") pod "node-exporter-h94cr" (UID: "4fab6402-db93-4b93-be4f-41eaaff1265e") : secret "node-exporter-tls" not found Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.409206 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-textfile\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.409479 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-wtmp\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.409512 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.409521 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-metrics-client-ca\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.410553 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.409858 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-accelerators-collector-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.411589 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.411567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.422179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.422125 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpf2q\" (UniqueName: \"kubernetes.io/projected/4fab6402-db93-4b93-be4f-41eaaff1265e-kube-api-access-qpf2q\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.808932 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.808830 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45678959-2d282" event={"ID":"c360507b-849f-4291-9b3c-11e77702d388","Type":"ContainerStarted","Data":"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8"} Jul 09 04:01:30.813427 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.813405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.815820 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.815800 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2c08bf2-554e-4749-8e40-084903a926e0-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mvt5n\" (UID: \"b2c08bf2-554e-4749-8e40-084903a926e0\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:30.829906 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.829858 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d45678959-2d282" podStartSLOduration=2.004355344 podStartE2EDuration="4.829845147s" totalImagesPullingTime="2.825489803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:30.82912351 +0000 UTC m=+76.944300558" watchObservedRunningTime="2026-07-09 04:01:30.829845147 +0000 UTC m=+76.945022182" Jul 09 04:01:30.914366 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.914323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:30.918323 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:30.918286 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fab6402-db93-4b93-be4f-41eaaff1265e-node-exporter-tls\") pod \"node-exporter-h94cr\" (UID: \"4fab6402-db93-4b93-be4f-41eaaff1265e\") " pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:31.049370 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:31.049336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" Jul 09 04:01:31.062076 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:31.062032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h94cr" Jul 09 04:01:31.072113 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:31.072085 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fab6402_db93_4b93_be4f_41eaaff1265e.slice/crio-e3b9a1658933955aad57eff8736327dc88d8e54fd0ee1e81e5a647a2635f6b66 WatchSource:0}: Error finding container e3b9a1658933955aad57eff8736327dc88d8e54fd0ee1e81e5a647a2635f6b66: Status 404 returned error can't find the container with id e3b9a1658933955aad57eff8736327dc88d8e54fd0ee1e81e5a647a2635f6b66 Jul 09 04:01:31.195987 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:31.195950 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n"] Jul 09 04:01:31.200991 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:01:31.200964 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c08bf2_554e_4749_8e40_084903a926e0.slice/crio-ec5bb81039d10480cce8e8726fd348536d52651a81e5df4c89b232f238440ceb WatchSource:0}: Error finding container ec5bb81039d10480cce8e8726fd348536d52651a81e5df4c89b232f238440ceb: Status 404 returned error can't find the container with id ec5bb81039d10480cce8e8726fd348536d52651a81e5df4c89b232f238440ceb Jul 09 04:01:31.813504 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:31.813465 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h94cr" event={"ID":"4fab6402-db93-4b93-be4f-41eaaff1265e","Type":"ContainerStarted","Data":"e3b9a1658933955aad57eff8736327dc88d8e54fd0ee1e81e5a647a2635f6b66"} Jul 09 04:01:31.814633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:31.814603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" event={"ID":"b2c08bf2-554e-4749-8e40-084903a926e0","Type":"ContainerStarted","Data":"ec5bb81039d10480cce8e8726fd348536d52651a81e5df4c89b232f238440ceb"} Jul 09 04:01:33.715240 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:33.715211 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64ddc67c5d-2vs8n" Jul 09 04:01:37.162692 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:37.162567 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:37.163138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:37.162899 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:37.169130 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:37.169104 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:37.845186 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:37.845159 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:01:41.857914 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.857870 2570 generic.go:356] "Generic (PLEG): container finished" podID="4fab6402-db93-4b93-be4f-41eaaff1265e" containerID="fb4833ae0f29044e39eefcee817a16d4a501e14e024dc16818c3e3ccc8f0c39a" exitCode=0 Jul 09 04:01:41.858378 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.857998 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h94cr" event={"ID":"4fab6402-db93-4b93-be4f-41eaaff1265e","Type":"ContainerDied","Data":"fb4833ae0f29044e39eefcee817a16d4a501e14e024dc16818c3e3ccc8f0c39a"} Jul 09 04:01:41.860278 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.860242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-f74hh" event={"ID":"bcd61866-271e-4b98-9092-63140cf94d74","Type":"ContainerStarted","Data":"0d2e20ac513b957b60c6264ff517110e6d8b0c7781fa3fcf6f3dcce436f70be0"} Jul 09 04:01:41.860513 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.860468 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:41.864292 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.864271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" event={"ID":"b2c08bf2-554e-4749-8e40-084903a926e0","Type":"ContainerStarted","Data":"b0f653899a1996959112e762f0f0eadcb55854f77309325528c58158949195d6"} Jul 09 04:01:41.864387 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.864317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" event={"ID":"b2c08bf2-554e-4749-8e40-084903a926e0","Type":"ContainerStarted","Data":"da14367fb877cf23a9ceca04ae271a7520c8230d7575a9979a2c642248513f78"} Jul 09 04:01:41.864387 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.864348 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" event={"ID":"b2c08bf2-554e-4749-8e40-084903a926e0","Type":"ContainerStarted","Data":"2ae99c268cbf03766efb126e60c11769fe68d252af1f7fa90536c1336740dc45"} Jul 09 04:01:41.872577 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.872555 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-f74hh" Jul 09 04:01:41.915970 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:41.915848 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mvt5n" podStartSLOduration=2.326562801 podStartE2EDuration="11.915823073s" totalImagesPullingTime="9.589260272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:41.896178508 +0000 UTC m=+88.011355570" watchObservedRunningTime="2026-07-09 04:01:41.915823073 +0000 UTC m=+88.031000110" Jul 09 04:01:42.871865 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:42.871780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h94cr" event={"ID":"4fab6402-db93-4b93-be4f-41eaaff1265e","Type":"ContainerStarted","Data":"82d7a4eac6421d4364fefd2c6728aac24564642bfeb64c09d51ca812ad38c7fc"} Jul 09 04:01:42.871865 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:42.871863 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h94cr" event={"ID":"4fab6402-db93-4b93-be4f-41eaaff1265e","Type":"ContainerStarted","Data":"a61cf4a1f1ff114ef84166e4d280aecbf4ec89109a5a8e903f2cd15987cb8fdd"} Jul 09 04:01:42.898411 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:42.898356 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-f74hh" podStartSLOduration=3.288470666 podStartE2EDuration="26.898336115s" totalImagesPullingTime="23.609865449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:41.917070827 +0000 UTC m=+88.032247859" watchObservedRunningTime="2026-07-09 04:01:42.898336115 +0000 UTC m=+89.013513153" Jul 09 04:01:42.898592 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:42.898468 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h94cr" podStartSLOduration=3.040023634 podStartE2EDuration="12.898460405s" totalImagesPullingTime="9.716845009s" totalInitContainerRuntime="141.591762ms" isStatefulPod=true podCreationTimestamp="2026-07-09 04:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:42.897031251 +0000 UTC m=+89.012208287" watchObservedRunningTime="2026-07-09 04:01:42.898460405 +0000 UTC m=+89.013637441" Jul 09 04:01:51.552994 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:51.552955 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:01:56.790519 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:01:56.790487 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fccbg" Jul 09 04:02:16.579343 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.579288 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d45678959-2d282" podUID="c360507b-849f-4291-9b3c-11e77702d388" containerName="console" containerID="cri-o://fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8" gracePeriod=15 Jul 09 04:02:16.851729 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.851662 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:02:16.986069 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986004 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986069 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986060 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkqp9\" (UniqueName: \"kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986318 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986087 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986318 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986104 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986318 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986125 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986318 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986168 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986318 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986253 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config\") pod \"c360507b-849f-4291-9b3c-11e77702d388\" (UID: \"c360507b-849f-4291-9b3c-11e77702d388\") " Jul 09 04:02:16.986674 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986568 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca" (OuterVolumeSpecName: "service-ca") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:16.986786 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986720 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config" (OuterVolumeSpecName: "console-config") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:16.986786 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986575 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:16.986786 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.986677 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:02:16.988499 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.988471 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:16.988587 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.988517 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:02:16.988587 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:16.988523 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9" (OuterVolumeSpecName: "kube-api-access-gkqp9") pod "c360507b-849f-4291-9b3c-11e77702d388" (UID: "c360507b-849f-4291-9b3c-11e77702d388"). InnerVolumeSpecName "kube-api-access-gkqp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:02:17.002483 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.002458 2570 generic.go:356] "Generic (PLEG): container finished" podID="c360507b-849f-4291-9b3c-11e77702d388" containerID="fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8" exitCode=0 Jul 09 04:02:17.002567 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.002501 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d45678959-2d282" Jul 09 04:02:17.002567 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.002534 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45678959-2d282" event={"ID":"c360507b-849f-4291-9b3c-11e77702d388","Type":"ContainerDied","Data":"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8"} Jul 09 04:02:17.002639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.002582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d45678959-2d282" event={"ID":"c360507b-849f-4291-9b3c-11e77702d388","Type":"ContainerDied","Data":"0d7b79183b914d2aef1fc99ed95852f6c042f40f0523d93a55109df84ac827a0"} Jul 09 04:02:17.002639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.002612 2570 scope.go:117] "RemoveContainer" containerID="fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8" Jul 09 04:02:17.011182 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.011165 2570 scope.go:117] "RemoveContainer" containerID="fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8" Jul 09 04:02:17.011450 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:02:17.011431 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8\": container with ID starting with fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8 not found: ID does not exist" containerID="fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8" Jul 09 04:02:17.011503 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.011459 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8"} err="failed to get container status \"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8\": rpc error: code = NotFound desc = could not find container \"fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8\": container with ID starting with fc79c3d655d23c7af7bcc36b282c223f29c772740fe65a6226eeec431c7531c8 not found: ID does not exist" Jul 09 04:02:17.024078 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.024037 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:02:17.029610 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.029593 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d45678959-2d282"] Jul 09 04:02:17.087290 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087261 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-console-config\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087290 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087284 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-serving-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087299 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqp9\" (UniqueName: \"kubernetes.io/projected/c360507b-849f-4291-9b3c-11e77702d388-kube-api-access-gkqp9\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087312 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c360507b-849f-4291-9b3c-11e77702d388-console-oauth-config\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087320 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-service-ca\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087328 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-trusted-ca-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:17.087421 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:17.087336 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c360507b-849f-4291-9b3c-11e77702d388-oauth-serving-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:02:18.426157 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:02:18.426125 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c360507b-849f-4291-9b3c-11e77702d388" path="/var/lib/kubelet/pods/c360507b-849f-4291-9b3c-11e77702d388/volumes" Jul 09 04:03:07.084061 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.083971 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:03:07.084578 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.084231 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c360507b-849f-4291-9b3c-11e77702d388" containerName="console" Jul 09 04:03:07.084578 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.084243 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c360507b-849f-4291-9b3c-11e77702d388" containerName="console" Jul 09 04:03:07.084578 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.084294 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c360507b-849f-4291-9b3c-11e77702d388" containerName="console" Jul 09 04:03:07.086105 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.086085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.088851 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.088830 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 04:03:07.090008 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.089993 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 04:03:07.090111 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.090009 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 04:03:07.090111 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.090028 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 04:03:07.090197 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.090106 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 04:03:07.090453 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.090432 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd6vl\"" Jul 09 04:03:07.094982 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.094965 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 04:03:07.100120 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.100103 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:03:07.152556 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152556 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152756 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152578 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152756 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152756 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152756 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.152900 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.152762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2tw\" (UniqueName: \"kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253163 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2tw\" (UniqueName: \"kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.253975 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253943 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.254284 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.253946 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.254284 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.254025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.254407 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.254265 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.255863 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.255841 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.256104 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.256084 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.262254 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.262232 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2tw\" (UniqueName: \"kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw\") pod \"console-7859884ffc-jv76z\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.395337 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.395255 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:07.523766 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:07.523735 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:03:07.526701 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:03:07.526676 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfccc6e6_fa9d_4221_9512_cb9f856adee2.slice/crio-7be9b1ffc95ba5ec655a645ffd9e9533f0b24029ebbfb2df2f7ea54c4f7d9fde WatchSource:0}: Error finding container 7be9b1ffc95ba5ec655a645ffd9e9533f0b24029ebbfb2df2f7ea54c4f7d9fde: Status 404 returned error can't find the container with id 7be9b1ffc95ba5ec655a645ffd9e9533f0b24029ebbfb2df2f7ea54c4f7d9fde Jul 09 04:03:08.176910 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:08.176877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7859884ffc-jv76z" event={"ID":"dfccc6e6-fa9d-4221-9512-cb9f856adee2","Type":"ContainerStarted","Data":"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b"} Jul 09 04:03:08.177350 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:08.176924 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7859884ffc-jv76z" event={"ID":"dfccc6e6-fa9d-4221-9512-cb9f856adee2","Type":"ContainerStarted","Data":"7be9b1ffc95ba5ec655a645ffd9e9533f0b24029ebbfb2df2f7ea54c4f7d9fde"} Jul 09 04:03:08.193970 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:08.193923 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7859884ffc-jv76z" podStartSLOduration=1.193909387 podStartE2EDuration="1.193909387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:03:08.193327725 +0000 UTC m=+174.308504761" watchObservedRunningTime="2026-07-09 04:03:08.193909387 +0000 UTC m=+174.309086423" Jul 09 04:03:17.396456 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:17.396418 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:17.396874 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:17.396483 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:17.401216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:17.401196 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:18.214929 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:18.214896 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:03:30.609006 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.608969 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m"] Jul 09 04:03:30.612202 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.612183 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.615265 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.615239 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 04:03:30.615423 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.615239 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 04:03:30.615565 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.615270 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sx9rf\"" Jul 09 04:03:30.621833 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.621810 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m"] Jul 09 04:03:30.727860 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.727830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.727860 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.727864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.728074 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.727883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qzv\" (UniqueName: \"kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.828340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.828309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.828493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.828350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.828493 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.828374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qzv\" (UniqueName: \"kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.828701 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.828683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.828763 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.828746 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.836487 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.836458 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qzv\" (UniqueName: \"kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:30.921658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:30.921629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:31.048112 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:31.048088 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m"] Jul 09 04:03:31.050772 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:03:31.050740 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc10d64_0b2b_48f4_b713_d9fb5b3e4a53.slice/crio-7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb WatchSource:0}: Error finding container 7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb: Status 404 returned error can't find the container with id 7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb Jul 09 04:03:31.256361 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:31.256281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" event={"ID":"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53","Type":"ContainerStarted","Data":"7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb"} Jul 09 04:03:39.289719 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:39.289680 2570 generic.go:356] "Generic (PLEG): container finished" podID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerID="2c12cdafa0034f394834e41921008285cbf840d19e6561d059c4ef1e3941186f" exitCode=0 Jul 09 04:03:39.290211 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:39.289769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" event={"ID":"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53","Type":"ContainerDied","Data":"2c12cdafa0034f394834e41921008285cbf840d19e6561d059c4ef1e3941186f"} Jul 09 04:03:42.302193 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:42.302149 2570 generic.go:356] "Generic (PLEG): container finished" podID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerID="1509dba635c984e95f29753bd640adeef889f40df68bc0b41ac9f05c6ad4db95" exitCode=0 Jul 09 04:03:42.302632 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:42.302226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" event={"ID":"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53","Type":"ContainerDied","Data":"1509dba635c984e95f29753bd640adeef889f40df68bc0b41ac9f05c6ad4db95"} Jul 09 04:03:53.355662 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:53.355590 2570 generic.go:356] "Generic (PLEG): container finished" podID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerID="bcf75f93255849c4fb67d3c75da3a1e65b3179b220169170996f7bb0ddbd1011" exitCode=0 Jul 09 04:03:53.355955 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:53.355679 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" event={"ID":"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53","Type":"ContainerDied","Data":"bcf75f93255849c4fb67d3c75da3a1e65b3179b220169170996f7bb0ddbd1011"} Jul 09 04:03:54.512082 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.512030 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:54.626797 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.626699 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle\") pod \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " Jul 09 04:03:54.626797 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.626795 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util\") pod \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " Jul 09 04:03:54.627012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.626829 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5qzv\" (UniqueName: \"kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv\") pod \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\" (UID: \"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53\") " Jul 09 04:03:54.627440 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.627407 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle" (OuterVolumeSpecName: "bundle") pod "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" (UID: "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:03:54.629081 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.629058 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv" (OuterVolumeSpecName: "kube-api-access-b5qzv") pod "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" (UID: "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53"). InnerVolumeSpecName "kube-api-access-b5qzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:03:54.631337 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.631312 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util" (OuterVolumeSpecName: "util") pod "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" (UID: "5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:03:54.728134 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.728102 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:03:54.728134 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.728129 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qzv\" (UniqueName: \"kubernetes.io/projected/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-kube-api-access-b5qzv\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:03:54.728134 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:54.728139 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:03:55.364290 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:55.364262 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" Jul 09 04:03:55.364290 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:55.364269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p666m" event={"ID":"5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53","Type":"ContainerDied","Data":"7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb"} Jul 09 04:03:55.364480 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:03:55.364310 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7498060cd49e2e7d209e297b4e2552b948da20b385f21150077e8227213961eb" Jul 09 04:04:03.613455 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613414 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8mmfs"] Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613684 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="extract" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613698 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="extract" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613715 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="util" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613722 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="util" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613727 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="pull" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613733 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="pull" Jul 09 04:04:03.613882 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.613780 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc10d64-0b2b-48f4-b713-d9fb5b3e4a53" containerName="extract" Jul 09 04:04:03.616295 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.616272 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.622546 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.622210 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 04:04:03.622546 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.622490 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 04:04:03.623614 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.623593 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4fnfc\"" Jul 09 04:04:03.629973 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.629953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8mmfs"] Jul 09 04:04:03.693469 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.693438 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd"] Jul 09 04:04:03.694483 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.694458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwps\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-kube-api-access-jpwps\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.694595 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.694531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.696019 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.696005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.705586 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.705568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sx9rf\"" Jul 09 04:04:03.705675 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.705662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 04:04:03.711348 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.711329 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 04:04:03.743742 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.743714 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd"] Jul 09 04:04:03.795694 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.795670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwps\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-kube-api-access-jpwps\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.795848 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.795719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzfv\" (UniqueName: \"kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.795925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.795841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.795925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.795906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.796091 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.795935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.815483 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.815456 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.818015 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.817992 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwps\" (UniqueName: \"kubernetes.io/projected/c315d9ca-3ea0-4a04-9a49-ad61a3f19144-kube-api-access-jpwps\") pod \"cert-manager-webhook-6486ffd997-8mmfs\" (UID: \"c315d9ca-3ea0-4a04-9a49-ad61a3f19144\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:03.896371 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.896285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzfv\" (UniqueName: \"kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.896371 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.896331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.896371 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.896349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.896638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.896623 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.896674 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.896661 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.912442 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.912415 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzfv\" (UniqueName: \"kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:03.925433 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:03.925414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:04.004363 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.004329 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:04.060843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.060813 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8mmfs"] Jul 09 04:04:04.066752 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:04.066726 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc315d9ca_3ea0_4a04_9a49_ad61a3f19144.slice/crio-cb58e5c473f6d5beef7658b071575b4197eb79f6fd0f94c361947d7e89add0f8 WatchSource:0}: Error finding container cb58e5c473f6d5beef7658b071575b4197eb79f6fd0f94c361947d7e89add0f8: Status 404 returned error can't find the container with id cb58e5c473f6d5beef7658b071575b4197eb79f6fd0f94c361947d7e89add0f8 Jul 09 04:04:04.137928 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.137832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd"] Jul 09 04:04:04.140411 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:04.140386 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ee2512_f626_4163_8d16_148bd13cdad3.slice/crio-1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084 WatchSource:0}: Error finding container 1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084: Status 404 returned error can't find the container with id 1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084 Jul 09 04:04:04.394998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.394960 2570 generic.go:356] "Generic (PLEG): container finished" podID="52ee2512-f626-4163-8d16-148bd13cdad3" containerID="cc3899e1dc2c5a1cbe2bc5b9ff5178c8bb3b9a0779672acb3e4cadae6834b64a" exitCode=0 Jul 09 04:04:04.395194 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.395001 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" event={"ID":"52ee2512-f626-4163-8d16-148bd13cdad3","Type":"ContainerDied","Data":"cc3899e1dc2c5a1cbe2bc5b9ff5178c8bb3b9a0779672acb3e4cadae6834b64a"} Jul 09 04:04:04.395194 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.395074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" event={"ID":"52ee2512-f626-4163-8d16-148bd13cdad3","Type":"ContainerStarted","Data":"1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084"} Jul 09 04:04:04.396413 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:04.396379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" event={"ID":"c315d9ca-3ea0-4a04-9a49-ad61a3f19144","Type":"ContainerStarted","Data":"cb58e5c473f6d5beef7658b071575b4197eb79f6fd0f94c361947d7e89add0f8"} Jul 09 04:04:08.413591 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:08.413556 2570 generic.go:356] "Generic (PLEG): container finished" podID="52ee2512-f626-4163-8d16-148bd13cdad3" containerID="d7d375a58a07101e2f7101a649e86c0d83c660fee14a60cab5a73ffc56a2e265" exitCode=0 Jul 09 04:04:08.414004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:08.413640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" event={"ID":"52ee2512-f626-4163-8d16-148bd13cdad3","Type":"ContainerDied","Data":"d7d375a58a07101e2f7101a649e86c0d83c660fee14a60cab5a73ffc56a2e265"} Jul 09 04:04:08.415190 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:08.415166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" event={"ID":"c315d9ca-3ea0-4a04-9a49-ad61a3f19144","Type":"ContainerStarted","Data":"5bebe2876df38221c836a3c9fd8d66313bba4efa59f69f25d209a40c389c34fb"} Jul 09 04:04:08.415291 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:08.415285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:08.451626 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:08.451581 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" podStartSLOduration=1.345214 podStartE2EDuration="5.451542013s" totalImagesPullingTime="4.106328013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:08.450617873 +0000 UTC m=+234.565794935" watchObservedRunningTime="2026-07-09 04:04:08.451542013 +0000 UTC m=+234.566719049" Jul 09 04:04:09.420646 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:09.420611 2570 generic.go:356] "Generic (PLEG): container finished" podID="52ee2512-f626-4163-8d16-148bd13cdad3" containerID="276488d562a519285e458fd41c17872cc9831fdd8dc2be69d091896e78131721" exitCode=0 Jul 09 04:04:09.421007 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:09.420645 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" event={"ID":"52ee2512-f626-4163-8d16-148bd13cdad3","Type":"ContainerDied","Data":"276488d562a519285e458fd41c17872cc9831fdd8dc2be69d091896e78131721"} Jul 09 04:04:10.550929 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.550907 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:10.655079 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.655034 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util\") pod \"52ee2512-f626-4163-8d16-148bd13cdad3\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " Jul 09 04:04:10.655257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.655100 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdzfv\" (UniqueName: \"kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv\") pod \"52ee2512-f626-4163-8d16-148bd13cdad3\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " Jul 09 04:04:10.655257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.655131 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle\") pod \"52ee2512-f626-4163-8d16-148bd13cdad3\" (UID: \"52ee2512-f626-4163-8d16-148bd13cdad3\") " Jul 09 04:04:10.655559 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.655531 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle" (OuterVolumeSpecName: "bundle") pod "52ee2512-f626-4163-8d16-148bd13cdad3" (UID: "52ee2512-f626-4163-8d16-148bd13cdad3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:04:10.657404 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.657379 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv" (OuterVolumeSpecName: "kube-api-access-mdzfv") pod "52ee2512-f626-4163-8d16-148bd13cdad3" (UID: "52ee2512-f626-4163-8d16-148bd13cdad3"). InnerVolumeSpecName "kube-api-access-mdzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:04:10.662282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.662241 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util" (OuterVolumeSpecName: "util") pod "52ee2512-f626-4163-8d16-148bd13cdad3" (UID: "52ee2512-f626-4163-8d16-148bd13cdad3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:04:10.756658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.756579 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:10.756658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.756605 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzfv\" (UniqueName: \"kubernetes.io/projected/52ee2512-f626-4163-8d16-148bd13cdad3-kube-api-access-mdzfv\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:10.756658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:10.756615 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52ee2512-f626-4163-8d16-148bd13cdad3-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:11.430435 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:11.430402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" event={"ID":"52ee2512-f626-4163-8d16-148bd13cdad3","Type":"ContainerDied","Data":"1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084"} Jul 09 04:04:11.430435 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:11.430439 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvw4rd" Jul 09 04:04:11.430633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:11.430446 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1569aee80b3a441ddcab87792487915d53dd1cd42e4e8a349fb7a4091ecd6084" Jul 09 04:04:14.426558 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:14.426528 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8mmfs" Jul 09 04:04:23.580395 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580355 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz"] Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580735 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="util" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580752 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="util" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580763 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="extract" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580770 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="extract" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580797 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="pull" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580803 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="pull" Jul 09 04:04:23.580957 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.580876 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ee2512-f626-4163-8d16-148bd13cdad3" containerName="extract" Jul 09 04:04:23.584570 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.584550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.587286 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.587258 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 04:04:23.587286 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.587282 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 04:04:23.588434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.588419 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sx9rf\"" Jul 09 04:04:23.592367 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.592347 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz"] Jul 09 04:04:23.645697 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.645667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmhs\" (UniqueName: \"kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.645869 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.645715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.645869 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.645771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.746277 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.746246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.746436 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.746304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmhs\" (UniqueName: \"kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.746436 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.746350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.746628 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.746609 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.746720 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.746704 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.756877 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.756852 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmhs\" (UniqueName: \"kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:23.894508 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:23.894419 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:24.026436 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:24.026402 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz"] Jul 09 04:04:24.029233 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:24.029203 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acecfd2_75f0_42b0_97d7_c673946b8c36.slice/crio-d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081 WatchSource:0}: Error finding container d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081: Status 404 returned error can't find the container with id d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081 Jul 09 04:04:24.479080 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:24.479028 2570 generic.go:356] "Generic (PLEG): container finished" podID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerID="4dd2aae2f80f1d102a2216b736f716c1f8fe6b761f5a70800cf3afc5083a9fb6" exitCode=0 Jul 09 04:04:24.479249 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:24.479101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" event={"ID":"7acecfd2-75f0-42b0-97d7-c673946b8c36","Type":"ContainerDied","Data":"4dd2aae2f80f1d102a2216b736f716c1f8fe6b761f5a70800cf3afc5083a9fb6"} Jul 09 04:04:24.479249 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:24.479151 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" event={"ID":"7acecfd2-75f0-42b0-97d7-c673946b8c36","Type":"ContainerStarted","Data":"d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081"} Jul 09 04:04:25.484321 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:25.484286 2570 generic.go:356] "Generic (PLEG): container finished" podID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerID="5e1046bc0ac29eee7c4bcb4ad57a93e44b458c5c728b4b3f1164f5022ba2267b" exitCode=0 Jul 09 04:04:25.484712 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:25.484377 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" event={"ID":"7acecfd2-75f0-42b0-97d7-c673946b8c36","Type":"ContainerDied","Data":"5e1046bc0ac29eee7c4bcb4ad57a93e44b458c5c728b4b3f1164f5022ba2267b"} Jul 09 04:04:26.489741 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:26.489711 2570 generic.go:356] "Generic (PLEG): container finished" podID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerID="b3e2fc3aaa445c5f79a173456cda743855b14d297b91d618d3fdd97685441f5c" exitCode=0 Jul 09 04:04:26.490143 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:26.489748 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" event={"ID":"7acecfd2-75f0-42b0-97d7-c673946b8c36","Type":"ContainerDied","Data":"b3e2fc3aaa445c5f79a173456cda743855b14d297b91d618d3fdd97685441f5c"} Jul 09 04:04:27.619036 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.619012 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:27.669484 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.669453 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle\") pod \"7acecfd2-75f0-42b0-97d7-c673946b8c36\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " Jul 09 04:04:27.669667 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.669528 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util\") pod \"7acecfd2-75f0-42b0-97d7-c673946b8c36\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " Jul 09 04:04:27.669667 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.669558 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnmhs\" (UniqueName: \"kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs\") pod \"7acecfd2-75f0-42b0-97d7-c673946b8c36\" (UID: \"7acecfd2-75f0-42b0-97d7-c673946b8c36\") " Jul 09 04:04:27.670166 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.670129 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle" (OuterVolumeSpecName: "bundle") pod "7acecfd2-75f0-42b0-97d7-c673946b8c36" (UID: "7acecfd2-75f0-42b0-97d7-c673946b8c36"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:04:27.671746 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.671720 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs" (OuterVolumeSpecName: "kube-api-access-wnmhs") pod "7acecfd2-75f0-42b0-97d7-c673946b8c36" (UID: "7acecfd2-75f0-42b0-97d7-c673946b8c36"). InnerVolumeSpecName "kube-api-access-wnmhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:04:27.678065 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.678024 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util" (OuterVolumeSpecName: "util") pod "7acecfd2-75f0-42b0-97d7-c673946b8c36" (UID: "7acecfd2-75f0-42b0-97d7-c673946b8c36"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:04:27.770099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.770005 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:27.770099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.770030 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acecfd2-75f0-42b0-97d7-c673946b8c36-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:27.770099 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:27.770056 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmhs\" (UniqueName: \"kubernetes.io/projected/7acecfd2-75f0-42b0-97d7-c673946b8c36-kube-api-access-wnmhs\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:04:28.499193 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:28.499162 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" Jul 09 04:04:28.499348 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:28.499166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnptxz" event={"ID":"7acecfd2-75f0-42b0-97d7-c673946b8c36","Type":"ContainerDied","Data":"d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081"} Jul 09 04:04:28.499348 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:28.499279 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7942dc83b0086a959e258bb076935a4da854668b2a2f305b9e16c2dc0de4081" Jul 09 04:04:38.391819 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.391735 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977"] Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392008 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="util" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392019 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="util" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392026 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="pull" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392032 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="pull" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392059 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="extract" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392065 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="extract" Jul 09 04:04:38.392280 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.392112 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7acecfd2-75f0-42b0-97d7-c673946b8c36" containerName="extract" Jul 09 04:04:38.393937 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.393922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.397909 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.397886 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 04:04:38.398033 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.397936 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 04:04:38.398033 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.397995 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qqrfg\"" Jul 09 04:04:38.398212 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.398198 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 04:04:38.398485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.398474 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 04:04:38.418529 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.418488 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977"] Jul 09 04:04:38.455397 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.455368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.455745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.455413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.455745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.455433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.455745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.455448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64cw\" (UniqueName: \"kubernetes.io/projected/9b1cc303-74fd-4bd4-abc5-2c93319ea408-kube-api-access-s64cw\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.455745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.455510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556358 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64cw\" (UniqueName: \"kubernetes.io/projected/9b1cc303-74fd-4bd4-abc5-2c93319ea408-kube-api-access-s64cw\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556526 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.556813 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.556768 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-manifests\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.558794 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.558776 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1cc303-74fd-4bd4-abc5-2c93319ea408-tmp\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.558921 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.558903 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-webhook-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.559128 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.559107 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1cc303-74fd-4bd4-abc5-2c93319ea408-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.565434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.565413 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64cw\" (UniqueName: \"kubernetes.io/projected/9b1cc303-74fd-4bd4-abc5-2c93319ea408-kube-api-access-s64cw\") pod \"opendatahub-operator-controller-manager-67dd5f9985-pr977\" (UID: \"9b1cc303-74fd-4bd4-abc5-2c93319ea408\") " pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.704203 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.704167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:38.841457 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:38.841428 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977"] Jul 09 04:04:38.844887 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:38.844860 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1cc303_74fd_4bd4_abc5_2c93319ea408.slice/crio-541d2f176e4425e2050dc60739f3ff32637b853b3ee939fc732f8906fc5e195d WatchSource:0}: Error finding container 541d2f176e4425e2050dc60739f3ff32637b853b3ee939fc732f8906fc5e195d: Status 404 returned error can't find the container with id 541d2f176e4425e2050dc60739f3ff32637b853b3ee939fc732f8906fc5e195d Jul 09 04:04:39.542865 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:39.542825 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" event={"ID":"9b1cc303-74fd-4bd4-abc5-2c93319ea408","Type":"ContainerStarted","Data":"541d2f176e4425e2050dc60739f3ff32637b853b3ee939fc732f8906fc5e195d"} Jul 09 04:04:42.557651 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:42.557617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" event={"ID":"9b1cc303-74fd-4bd4-abc5-2c93319ea408","Type":"ContainerStarted","Data":"58363db48f6972869668d5843fbd7a1b863e300b8607d51c55255fc597e93d5d"} Jul 09 04:04:43.563468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:43.563435 2570 generic.go:356] "Generic (PLEG): container finished" podID="9b1cc303-74fd-4bd4-abc5-2c93319ea408" containerID="58363db48f6972869668d5843fbd7a1b863e300b8607d51c55255fc597e93d5d" exitCode=0 Jul 09 04:04:43.563468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:43.563472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" event={"ID":"9b1cc303-74fd-4bd4-abc5-2c93319ea408","Type":"ContainerDied","Data":"58363db48f6972869668d5843fbd7a1b863e300b8607d51c55255fc597e93d5d"} Jul 09 04:04:44.569673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:44.569632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" event={"ID":"9b1cc303-74fd-4bd4-abc5-2c93319ea408","Type":"ContainerStarted","Data":"e37bb8c0b622f4868cd8755a66635a355c2db43150475c6fb229b73a65194a3d"} Jul 09 04:04:44.570078 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:44.569753 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:44.595137 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:44.595090 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" podStartSLOduration=2.373692827 podStartE2EDuration="6.59507408s" totalImagesPullingTime="3.186447095s" totalInitContainerRuntime="1.034934158s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:04:44.593851339 +0000 UTC m=+270.709028385" watchObservedRunningTime="2026-07-09 04:04:44.59507408 +0000 UTC m=+270.710251115" Jul 09 04:04:55.576199 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:55.576167 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67dd5f9985-pr977" Jul 09 04:04:56.831943 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.831909 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb"] Jul 09 04:04:56.841591 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.841571 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:56.845061 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.844983 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 04:04:56.845061 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.844983 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 04:04:56.845308 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.845288 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 04:04:56.845356 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.845311 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bd226\"" Jul 09 04:04:56.845403 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.845380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 04:04:56.845814 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.845793 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 04:04:56.846271 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.846247 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb"] Jul 09 04:04:56.906841 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.906812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:56.906998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.906847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-metrics-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:56.906998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.906871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e6c8439-a172-4639-a898-f2ac7bff3557-manager-config\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:56.906998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:56.906897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0e6c8439-a172-4639-a898-f2ac7bff3557-kube-api-access-bnqvw\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.007922 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.007896 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.008092 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.007929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-metrics-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.008092 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.007954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e6c8439-a172-4639-a898-f2ac7bff3557-manager-config\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.008092 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.007983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0e6c8439-a172-4639-a898-f2ac7bff3557-kube-api-access-bnqvw\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.008689 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.008667 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e6c8439-a172-4639-a898-f2ac7bff3557-manager-config\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.010459 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.010434 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.010546 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.010469 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6c8439-a172-4639-a898-f2ac7bff3557-metrics-cert\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.023729 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.023682 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0e6c8439-a172-4639-a898-f2ac7bff3557-kube-api-access-bnqvw\") pod \"lws-controller-manager-ff67c745d-hj2xb\" (UID: \"0e6c8439-a172-4639-a898-f2ac7bff3557\") " pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.151928 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.151903 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:04:57.295933 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.295914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb"] Jul 09 04:04:57.298197 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:57.298171 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6c8439_a172_4639_a898_f2ac7bff3557.slice/crio-2f9dcd315492da626dfa42d28defa4737d406d4885e93e1c96bd221286be4d25 WatchSource:0}: Error finding container 2f9dcd315492da626dfa42d28defa4737d406d4885e93e1c96bd221286be4d25: Status 404 returned error can't find the container with id 2f9dcd315492da626dfa42d28defa4737d406d4885e93e1c96bd221286be4d25 Jul 09 04:04:57.623943 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:57.623855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" event={"ID":"0e6c8439-a172-4639-a898-f2ac7bff3557","Type":"ContainerStarted","Data":"2f9dcd315492da626dfa42d28defa4737d406d4885e93e1c96bd221286be4d25"} Jul 09 04:04:58.037577 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.037549 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm"] Jul 09 04:04:58.041847 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.041831 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.046143 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.046122 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 04:04:58.047260 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.047222 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 04:04:58.047460 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.047282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 04:04:58.047647 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.047630 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 04:04:58.047732 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.047662 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fchsg\"" Jul 09 04:04:58.060650 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.060631 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm"] Jul 09 04:04:58.117094 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.117072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkk7v\" (UniqueName: \"kubernetes.io/projected/087b5914-a55e-4c86-8503-303121798881-kube-api-access-lkk7v\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.117234 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.117110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087b5914-a55e-4c86-8503-303121798881-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.117234 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.117177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b5914-a55e-4c86-8503-303121798881-tmp\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.217764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.217736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b5914-a55e-4c86-8503-303121798881-tmp\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.217939 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.217780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkk7v\" (UniqueName: \"kubernetes.io/projected/087b5914-a55e-4c86-8503-303121798881-kube-api-access-lkk7v\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.217939 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.217806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087b5914-a55e-4c86-8503-303121798881-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.220189 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.220167 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b5914-a55e-4c86-8503-303121798881-tmp\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.220400 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.220377 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087b5914-a55e-4c86-8503-303121798881-tls-certs\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.233666 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.233642 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkk7v\" (UniqueName: \"kubernetes.io/projected/087b5914-a55e-4c86-8503-303121798881-kube-api-access-lkk7v\") pod \"kube-auth-proxy-7b68b5bb4-tq2gm\" (UID: \"087b5914-a55e-4c86-8503-303121798881\") " pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.351420 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.351343 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" Jul 09 04:04:58.495418 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.495392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm"] Jul 09 04:04:58.498083 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:04:58.498031 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b5914_a55e_4c86_8503_303121798881.slice/crio-12b298696030a7a2c408874ed6000ad6a5a63bd213fec34de343692f9e5fd5dc WatchSource:0}: Error finding container 12b298696030a7a2c408874ed6000ad6a5a63bd213fec34de343692f9e5fd5dc: Status 404 returned error can't find the container with id 12b298696030a7a2c408874ed6000ad6a5a63bd213fec34de343692f9e5fd5dc Jul 09 04:04:58.628919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:04:58.628823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" event={"ID":"087b5914-a55e-4c86-8503-303121798881","Type":"ContainerStarted","Data":"12b298696030a7a2c408874ed6000ad6a5a63bd213fec34de343692f9e5fd5dc"} Jul 09 04:05:01.642761 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:01.642693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" event={"ID":"087b5914-a55e-4c86-8503-303121798881","Type":"ContainerStarted","Data":"40318c2460763a7660d64985b997dd567b5b32ad7df434a4a709d5a4aeddaeb6"} Jul 09 04:05:01.661376 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:01.661323 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b68b5bb4-tq2gm" podStartSLOduration=0.838434864 podStartE2EDuration="3.661306562s" totalImagesPullingTime="2.822871698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:01.659938493 +0000 UTC m=+287.775115536" watchObservedRunningTime="2026-07-09 04:05:01.661306562 +0000 UTC m=+287.776483610" Jul 09 04:05:02.648080 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:02.648018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" event={"ID":"0e6c8439-a172-4639-a898-f2ac7bff3557","Type":"ContainerStarted","Data":"f07908f39f67a4bd26b45ec82654985ab4b6d0d8fff004fc7a9c22630442b95e"} Jul 09 04:05:02.648502 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:02.648183 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:05:02.664460 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:02.664420 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" podStartSLOduration=1.924632468 podStartE2EDuration="6.664405788s" totalImagesPullingTime="4.73977332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:02.663846928 +0000 UTC m=+288.779023968" watchObservedRunningTime="2026-07-09 04:05:02.664405788 +0000 UTC m=+288.779582824" Jul 09 04:05:13.654500 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:13.654465 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ff67c745d-hj2xb" Jul 09 04:05:14.294315 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:14.294287 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:05:14.294518 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:14.294500 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:05:14.297673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:05:14.297650 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 04:06:10.820922 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.820731 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh"] Jul 09 04:06:10.823412 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.823390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:10.826034 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.826004 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ltzz2\"" Jul 09 04:06:10.826170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.826006 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 04:06:10.827145 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.827130 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 04:06:10.831319 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.831298 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh"] Jul 09 04:06:10.903514 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.903492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:10.903647 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.903535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:10.903647 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:10.903558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7h9\" (UniqueName: \"kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.004188 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.004155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.004359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.004202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.004359 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.004228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7h9\" (UniqueName: \"kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.004540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.004518 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.004624 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.004605 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.012316 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.012296 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7h9\" (UniqueName: \"kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.133588 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.133508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:11.265204 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.265187 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh"] Jul 09 04:06:11.267412 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:06:11.267383 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c15c87_5c92_485a_b902_ac3e0381639d.slice/crio-65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba WatchSource:0}: Error finding container 65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba: Status 404 returned error can't find the container with id 65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba Jul 09 04:06:11.414583 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.414549 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx"] Jul 09 04:06:11.416914 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.416897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.425864 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.425840 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx"] Jul 09 04:06:11.508913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.508874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.509093 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.508968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.509093 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.508996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465zg\" (UniqueName: \"kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.610060 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.609998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.610229 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.610127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.610229 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.610169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465zg\" (UniqueName: \"kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.610481 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.610459 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.610516 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.610466 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.619896 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.619871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465zg\" (UniqueName: \"kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.741137 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.741060 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:11.864805 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.864782 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx"] Jul 09 04:06:11.867102 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:06:11.867076 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e88bca_f817_444e_8bc2_90244f80b2d9.slice/crio-ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b WatchSource:0}: Error finding container ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b: Status 404 returned error can't find the container with id ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b Jul 09 04:06:11.925168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.925132 2570 generic.go:356] "Generic (PLEG): container finished" podID="72c15c87-5c92-485a-b902-ac3e0381639d" containerID="4a1c20d236782364e43546ceaf7d51560f2061f1c07923d9a3685816ca806dbf" exitCode=0 Jul 09 04:06:11.925311 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.925213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" event={"ID":"72c15c87-5c92-485a-b902-ac3e0381639d","Type":"ContainerDied","Data":"4a1c20d236782364e43546ceaf7d51560f2061f1c07923d9a3685816ca806dbf"} Jul 09 04:06:11.925311 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.925267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" event={"ID":"72c15c87-5c92-485a-b902-ac3e0381639d","Type":"ContainerStarted","Data":"65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba"} Jul 09 04:06:11.925728 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.925711 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:06:11.926954 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.926927 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerStarted","Data":"272cb8210018bf6d2e1db2f4c54db483ac2fc65304883527f03388f81c171b4a"} Jul 09 04:06:11.927060 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:11.926975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerStarted","Data":"ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b"} Jul 09 04:06:12.034129 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.034102 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt"] Jul 09 04:06:12.036379 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.036364 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.046296 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.046273 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt"] Jul 09 04:06:12.113821 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.113785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jp6q\" (UniqueName: \"kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.113821 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.113828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.114067 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.113854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.214375 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.214341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp6q\" (UniqueName: \"kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.214558 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.214386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.214558 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.214411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.214748 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.214730 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.214813 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.214769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.223623 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.223597 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp6q\" (UniqueName: \"kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.345896 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.345804 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:12.481093 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.481070 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt"] Jul 09 04:06:12.483316 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:06:12.483291 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbf64a9_a617_4e38_9343_ca71175a10a3.slice/crio-40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d WatchSource:0}: Error finding container 40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d: Status 404 returned error can't find the container with id 40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d Jul 09 04:06:12.616301 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.616227 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57"] Jul 09 04:06:12.618656 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.618640 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.627364 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.627339 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57"] Jul 09 04:06:12.719004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.718968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.719004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.719004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88x82\" (UniqueName: \"kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.719237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.719071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.819619 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.819588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.819763 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.819646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.819763 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.819674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88x82\" (UniqueName: \"kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.819969 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.819948 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.820010 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.819991 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.828239 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.828210 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88x82\" (UniqueName: \"kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.927698 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.927671 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:12.933401 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.933368 2570 generic.go:356] "Generic (PLEG): container finished" podID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerID="f262a54b0309994b9e6656ffd2d7b7de7d68cccb09949be0aa07af199c55237c" exitCode=0 Jul 09 04:06:12.933505 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.933462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" event={"ID":"fdbf64a9-a617-4e38-9343-ca71175a10a3","Type":"ContainerDied","Data":"f262a54b0309994b9e6656ffd2d7b7de7d68cccb09949be0aa07af199c55237c"} Jul 09 04:06:12.933562 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.933514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" event={"ID":"fdbf64a9-a617-4e38-9343-ca71175a10a3","Type":"ContainerStarted","Data":"40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d"} Jul 09 04:06:12.935443 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.935425 2570 generic.go:356] "Generic (PLEG): container finished" podID="72c15c87-5c92-485a-b902-ac3e0381639d" containerID="a65d2ae0275a636ef1f72784e4b1f74841a3012f52d2058f6016ff824a59fbdb" exitCode=0 Jul 09 04:06:12.935534 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.935508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" event={"ID":"72c15c87-5c92-485a-b902-ac3e0381639d","Type":"ContainerDied","Data":"a65d2ae0275a636ef1f72784e4b1f74841a3012f52d2058f6016ff824a59fbdb"} Jul 09 04:06:12.937183 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.937161 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerID="272cb8210018bf6d2e1db2f4c54db483ac2fc65304883527f03388f81c171b4a" exitCode=0 Jul 09 04:06:12.937285 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:12.937195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerDied","Data":"272cb8210018bf6d2e1db2f4c54db483ac2fc65304883527f03388f81c171b4a"} Jul 09 04:06:13.106668 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.106639 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57"] Jul 09 04:06:13.108777 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:06:13.108749 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d942af_adc1_4508_9c8a_8da8eea0cfdb.slice/crio-7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7 WatchSource:0}: Error finding container 7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7: Status 404 returned error can't find the container with id 7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7 Jul 09 04:06:13.948598 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.948566 2570 generic.go:356] "Generic (PLEG): container finished" podID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerID="17f6228130d4857ead2305482eddf218ff306a9d51d04494fdcb859da54c0710" exitCode=0 Jul 09 04:06:13.949027 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.948650 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" event={"ID":"fdbf64a9-a617-4e38-9343-ca71175a10a3","Type":"ContainerDied","Data":"17f6228130d4857ead2305482eddf218ff306a9d51d04494fdcb859da54c0710"} Jul 09 04:06:13.950919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.950893 2570 generic.go:356] "Generic (PLEG): container finished" podID="72c15c87-5c92-485a-b902-ac3e0381639d" containerID="8ea5c6fab3bd6168e2a5259f702a8b6e1f82790b9d78cdb617460383273e3698" exitCode=0 Jul 09 04:06:13.951021 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.950972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" event={"ID":"72c15c87-5c92-485a-b902-ac3e0381639d","Type":"ContainerDied","Data":"8ea5c6fab3bd6168e2a5259f702a8b6e1f82790b9d78cdb617460383273e3698"} Jul 09 04:06:13.952703 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.952683 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerID="846adcb0fc4c32d8751face832f69c84b37f1221e2f81c44a6bb3bd9f100c0e0" exitCode=0 Jul 09 04:06:13.952795 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.952761 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerDied","Data":"846adcb0fc4c32d8751face832f69c84b37f1221e2f81c44a6bb3bd9f100c0e0"} Jul 09 04:06:13.954216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.954174 2570 generic.go:356] "Generic (PLEG): container finished" podID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerID="205dbb88c40f8b5d88be345302e3757a840cfc88620c0d62fef0ae01a3fdb3ad" exitCode=0 Jul 09 04:06:13.954216 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.954202 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" event={"ID":"89d942af-adc1-4508-9c8a-8da8eea0cfdb","Type":"ContainerDied","Data":"205dbb88c40f8b5d88be345302e3757a840cfc88620c0d62fef0ae01a3fdb3ad"} Jul 09 04:06:13.954315 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:13.954233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" event={"ID":"89d942af-adc1-4508-9c8a-8da8eea0cfdb","Type":"ContainerStarted","Data":"7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7"} Jul 09 04:06:14.960633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.960537 2570 generic.go:356] "Generic (PLEG): container finished" podID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerID="25fcb215fd7cfd245be1bbea73d1c6f58ab11495d0469e29f13944f6e64dc8bd" exitCode=0 Jul 09 04:06:14.960980 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.960622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" event={"ID":"fdbf64a9-a617-4e38-9343-ca71175a10a3","Type":"ContainerDied","Data":"25fcb215fd7cfd245be1bbea73d1c6f58ab11495d0469e29f13944f6e64dc8bd"} Jul 09 04:06:14.962585 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.962557 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerID="5c97650f7f872eefbd8dbe624ed646ed2a3c5bebbf06b043094def048bf72c94" exitCode=0 Jul 09 04:06:14.962696 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.962648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerDied","Data":"5c97650f7f872eefbd8dbe624ed646ed2a3c5bebbf06b043094def048bf72c94"} Jul 09 04:06:14.964340 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.964319 2570 generic.go:356] "Generic (PLEG): container finished" podID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerID="a5b254c002dbf3bf1923dce94708d3f8af129d397253cc74787a12a9121a9a9b" exitCode=0 Jul 09 04:06:14.964440 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:14.964383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" event={"ID":"89d942af-adc1-4508-9c8a-8da8eea0cfdb","Type":"ContainerDied","Data":"a5b254c002dbf3bf1923dce94708d3f8af129d397253cc74787a12a9121a9a9b"} Jul 09 04:06:15.108231 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.108205 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:15.243138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.243009 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle\") pod \"72c15c87-5c92-485a-b902-ac3e0381639d\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " Jul 09 04:06:15.243138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.243133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util\") pod \"72c15c87-5c92-485a-b902-ac3e0381639d\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " Jul 09 04:06:15.243353 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.243152 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k7h9\" (UniqueName: \"kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9\") pod \"72c15c87-5c92-485a-b902-ac3e0381639d\" (UID: \"72c15c87-5c92-485a-b902-ac3e0381639d\") " Jul 09 04:06:15.243578 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.243553 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle" (OuterVolumeSpecName: "bundle") pod "72c15c87-5c92-485a-b902-ac3e0381639d" (UID: "72c15c87-5c92-485a-b902-ac3e0381639d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:15.245581 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.245552 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9" (OuterVolumeSpecName: "kube-api-access-7k7h9") pod "72c15c87-5c92-485a-b902-ac3e0381639d" (UID: "72c15c87-5c92-485a-b902-ac3e0381639d"). InnerVolumeSpecName "kube-api-access-7k7h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:15.248523 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.248506 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util" (OuterVolumeSpecName: "util") pod "72c15c87-5c92-485a-b902-ac3e0381639d" (UID: "72c15c87-5c92-485a-b902-ac3e0381639d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:15.349205 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.344668 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:15.349205 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.344713 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c15c87-5c92-485a-b902-ac3e0381639d-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:15.349205 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.344728 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7h9\" (UniqueName: \"kubernetes.io/projected/72c15c87-5c92-485a-b902-ac3e0381639d-kube-api-access-7k7h9\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:15.970674 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.970633 2570 generic.go:356] "Generic (PLEG): container finished" podID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerID="ea1016ce4f443f5a220360c8f07a43ee467c475eebc417a693506f1bbf4b7f53" exitCode=0 Jul 09 04:06:15.971086 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.970724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" event={"ID":"89d942af-adc1-4508-9c8a-8da8eea0cfdb","Type":"ContainerDied","Data":"ea1016ce4f443f5a220360c8f07a43ee467c475eebc417a693506f1bbf4b7f53"} Jul 09 04:06:15.972494 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.972474 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" Jul 09 04:06:15.972494 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.972481 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh" event={"ID":"72c15c87-5c92-485a-b902-ac3e0381639d","Type":"ContainerDied","Data":"65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba"} Jul 09 04:06:15.972661 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:15.972524 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65660b4acca4f24886dbc891f94fbcc4c29f85555b5386a39ca609364c864dba" Jul 09 04:06:16.102268 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.102245 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:16.136617 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.136593 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:16.251606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251519 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util\") pod \"fdbf64a9-a617-4e38-9343-ca71175a10a3\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " Jul 09 04:06:16.251606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util\") pod \"00e88bca-f817-444e-8bc2-90244f80b2d9\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " Jul 09 04:06:16.251606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle\") pod \"fdbf64a9-a617-4e38-9343-ca71175a10a3\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " Jul 09 04:06:16.251892 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251626 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle\") pod \"00e88bca-f817-444e-8bc2-90244f80b2d9\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " Jul 09 04:06:16.251892 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251655 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-465zg\" (UniqueName: \"kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg\") pod \"00e88bca-f817-444e-8bc2-90244f80b2d9\" (UID: \"00e88bca-f817-444e-8bc2-90244f80b2d9\") " Jul 09 04:06:16.251892 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.251681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jp6q\" (UniqueName: \"kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q\") pod \"fdbf64a9-a617-4e38-9343-ca71175a10a3\" (UID: \"fdbf64a9-a617-4e38-9343-ca71175a10a3\") " Jul 09 04:06:16.252228 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.252189 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle" (OuterVolumeSpecName: "bundle") pod "fdbf64a9-a617-4e38-9343-ca71175a10a3" (UID: "fdbf64a9-a617-4e38-9343-ca71175a10a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:16.252363 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.252228 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle" (OuterVolumeSpecName: "bundle") pod "00e88bca-f817-444e-8bc2-90244f80b2d9" (UID: "00e88bca-f817-444e-8bc2-90244f80b2d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:16.253952 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.253924 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q" (OuterVolumeSpecName: "kube-api-access-2jp6q") pod "fdbf64a9-a617-4e38-9343-ca71175a10a3" (UID: "fdbf64a9-a617-4e38-9343-ca71175a10a3"). InnerVolumeSpecName "kube-api-access-2jp6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:16.254106 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.254069 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg" (OuterVolumeSpecName: "kube-api-access-465zg") pod "00e88bca-f817-444e-8bc2-90244f80b2d9" (UID: "00e88bca-f817-444e-8bc2-90244f80b2d9"). InnerVolumeSpecName "kube-api-access-465zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:16.257343 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.257321 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util" (OuterVolumeSpecName: "util") pod "fdbf64a9-a617-4e38-9343-ca71175a10a3" (UID: "fdbf64a9-a617-4e38-9343-ca71175a10a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:16.257587 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.257569 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util" (OuterVolumeSpecName: "util") pod "00e88bca-f817-444e-8bc2-90244f80b2d9" (UID: "00e88bca-f817-444e-8bc2-90244f80b2d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:16.353168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353139 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.353168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353164 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.353168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353173 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbf64a9-a617-4e38-9343-ca71175a10a3-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.353424 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353182 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00e88bca-f817-444e-8bc2-90244f80b2d9-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.353424 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353191 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465zg\" (UniqueName: \"kubernetes.io/projected/00e88bca-f817-444e-8bc2-90244f80b2d9-kube-api-access-465zg\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.353424 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.353201 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jp6q\" (UniqueName: \"kubernetes.io/projected/fdbf64a9-a617-4e38-9343-ca71175a10a3-kube-api-access-2jp6q\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:16.978596 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.978566 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" Jul 09 04:06:16.979082 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.978596 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx" event={"ID":"00e88bca-f817-444e-8bc2-90244f80b2d9","Type":"ContainerDied","Data":"ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b"} Jul 09 04:06:16.979082 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.978666 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd544e0b293e8883372e3cb23a05b4f7d791f8ece2cc30fc0b57cced881e30b" Jul 09 04:06:16.980515 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.980488 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" Jul 09 04:06:16.980515 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.980493 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt" event={"ID":"fdbf64a9-a617-4e38-9343-ca71175a10a3","Type":"ContainerDied","Data":"40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d"} Jul 09 04:06:16.980704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:16.980543 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e201fdca9dcb888dd1638affedc0999ff9f31492cb4d87572fd857bc878e2d" Jul 09 04:06:17.108209 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.108189 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:17.261775 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.261681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88x82\" (UniqueName: \"kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82\") pod \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " Jul 09 04:06:17.261775 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.261763 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle\") pod \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " Jul 09 04:06:17.261963 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.261792 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util\") pod \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\" (UID: \"89d942af-adc1-4508-9c8a-8da8eea0cfdb\") " Jul 09 04:06:17.262384 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.262352 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle" (OuterVolumeSpecName: "bundle") pod "89d942af-adc1-4508-9c8a-8da8eea0cfdb" (UID: "89d942af-adc1-4508-9c8a-8da8eea0cfdb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:17.263924 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.263903 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82" (OuterVolumeSpecName: "kube-api-access-88x82") pod "89d942af-adc1-4508-9c8a-8da8eea0cfdb" (UID: "89d942af-adc1-4508-9c8a-8da8eea0cfdb"). InnerVolumeSpecName "kube-api-access-88x82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:06:17.267031 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.266998 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util" (OuterVolumeSpecName: "util") pod "89d942af-adc1-4508-9c8a-8da8eea0cfdb" (UID: "89d942af-adc1-4508-9c8a-8da8eea0cfdb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:06:17.362666 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.362621 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:17.362666 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.362659 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d942af-adc1-4508-9c8a-8da8eea0cfdb-util\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:17.362666 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.362667 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88x82\" (UniqueName: \"kubernetes.io/projected/89d942af-adc1-4508-9c8a-8da8eea0cfdb-kube-api-access-88x82\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:06:17.986807 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.986779 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" Jul 09 04:06:17.986807 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.986780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57" event={"ID":"89d942af-adc1-4508-9c8a-8da8eea0cfdb","Type":"ContainerDied","Data":"7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7"} Jul 09 04:06:17.987235 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:17.986828 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b97baa31323f1480f36d5b76b9487e0996c502e50182b7cf7e687ec2c1146e7" Jul 09 04:06:39.233447 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233404 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl"] Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233779 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233793 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233801 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="extract" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233806 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="extract" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233813 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233818 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233832 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233840 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233846 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233851 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="pull" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233861 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233865 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233875 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233879 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="util" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233885 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="extract" Jul 09 04:06:39.233886 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233889 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233896 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="util" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233901 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="util" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233906 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233910 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233919 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233923 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233928 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="pull" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233933 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="pull" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233983 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c15c87-5c92-485a-b902-ac3e0381639d" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233993 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdbf64a9-a617-4e38-9343-ca71175a10a3" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.233998 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e88bca-f817-444e-8bc2-90244f80b2d9" containerName="extract" Jul 09 04:06:39.234402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.234003 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d942af-adc1-4508-9c8a-8da8eea0cfdb" containerName="extract" Jul 09 04:06:39.236806 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.236792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.239500 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.239471 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 04:06:39.239639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.239474 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ltzz2\"" Jul 09 04:06:39.239639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.239474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 04:06:39.239639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.239546 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 04:06:39.240642 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.240623 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 04:06:39.246312 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.246282 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl"] Jul 09 04:06:39.339264 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.339226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kms8z\" (UniqueName: \"kubernetes.io/projected/eb79263d-966a-4b1c-a3fd-6a158440a519-kube-api-access-kms8z\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.339264 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.339266 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb79263d-966a-4b1c-a3fd-6a158440a519-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.339484 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.339293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.440260 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.440228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms8z\" (UniqueName: \"kubernetes.io/projected/eb79263d-966a-4b1c-a3fd-6a158440a519-kube-api-access-kms8z\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.440260 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.440262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb79263d-966a-4b1c-a3fd-6a158440a519-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.440481 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.440460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.440579 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:06:39.440552 2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 09 04:06:39.440704 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:06:39.440631 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert podName:eb79263d-966a-4b1c-a3fd-6a158440a519 nodeName:}" failed. No retries permitted until 2026-07-09 04:06:39.940611685 +0000 UTC m=+386.055788699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-88fvl" (UID: "eb79263d-966a-4b1c-a3fd-6a158440a519") : secret "plugin-serving-cert" not found Jul 09 04:06:39.440914 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.440897 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb79263d-966a-4b1c-a3fd-6a158440a519-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.451484 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.451452 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms8z\" (UniqueName: \"kubernetes.io/projected/eb79263d-966a-4b1c-a3fd-6a158440a519-kube-api-access-kms8z\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.945857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.945828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:39.948332 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:39.948303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb79263d-966a-4b1c-a3fd-6a158440a519-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88fvl\" (UID: \"eb79263d-966a-4b1c-a3fd-6a158440a519\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:40.147657 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:40.147621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" Jul 09 04:06:40.276436 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:40.276413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl"] Jul 09 04:06:40.278506 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:06:40.278477 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb79263d_966a_4b1c_a3fd_6a158440a519.slice/crio-090208b9ddd72eea7c3adecb6fcb87187e46996dbe982ce306de054e034a05f7 WatchSource:0}: Error finding container 090208b9ddd72eea7c3adecb6fcb87187e46996dbe982ce306de054e034a05f7: Status 404 returned error can't find the container with id 090208b9ddd72eea7c3adecb6fcb87187e46996dbe982ce306de054e034a05f7 Jul 09 04:06:41.095932 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:06:41.095901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" event={"ID":"eb79263d-966a-4b1c-a3fd-6a158440a519","Type":"ContainerStarted","Data":"090208b9ddd72eea7c3adecb6fcb87187e46996dbe982ce306de054e034a05f7"} Jul 09 04:07:04.213511 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:04.213476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" event={"ID":"eb79263d-966a-4b1c-a3fd-6a158440a519","Type":"ContainerStarted","Data":"63e5be7b357d353900e8fdfc7875120154db25b13a9d7fa8cde05c434fe9972c"} Jul 09 04:07:04.231157 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:04.231112 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88fvl" podStartSLOduration=1.729754974 podStartE2EDuration="25.231098222s" totalImagesPullingTime="23.501343248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:04.229140521 +0000 UTC m=+410.344317558" watchObservedRunningTime="2026-07-09 04:07:04.231098222 +0000 UTC m=+410.346275257" Jul 09 04:07:11.102396 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:11.102359 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:07:25.779732 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:25.779696 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:07:25.948992 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:25.948955 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:07:25.948992 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:25.948993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:07:25.949222 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:25.949097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:25.953622 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:25.953598 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 04:07:26.053070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.052964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4qq\" (UniqueName: \"kubernetes.io/projected/8c0d9744-f573-4a22-bff5-ab017b94c7bb-kube-api-access-sh4qq\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.053070 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.053027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c0d9744-f573-4a22-bff5-ab017b94c7bb-config-file\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.153798 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.153762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4qq\" (UniqueName: \"kubernetes.io/projected/8c0d9744-f573-4a22-bff5-ab017b94c7bb-kube-api-access-sh4qq\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.153970 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.153818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c0d9744-f573-4a22-bff5-ab017b94c7bb-config-file\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.154406 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.154390 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c0d9744-f573-4a22-bff5-ab017b94c7bb-config-file\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.185151 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.185125 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4qq\" (UniqueName: \"kubernetes.io/projected/8c0d9744-f573-4a22-bff5-ab017b94c7bb-kube-api-access-sh4qq\") pod \"limitador-limitador-54d6d988d5-m5c6b\" (UID: \"8c0d9744-f573-4a22-bff5-ab017b94c7bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.188397 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.188371 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:26.211375 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.211342 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:26.211550 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.211454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:26.216796 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.216773 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hfbcf\"" Jul 09 04:07:26.254572 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.254542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47zm\" (UniqueName: \"kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm\") pod \"authorino-58cd955675-7dl7s\" (UID: \"4465214e-083c-417c-b2c3-d9730ecfd403\") " pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:26.258515 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.258485 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:26.355570 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.355537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47zm\" (UniqueName: \"kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm\") pod \"authorino-58cd955675-7dl7s\" (UID: \"4465214e-083c-417c-b2c3-d9730ecfd403\") " pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:26.364952 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.364914 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47zm\" (UniqueName: \"kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm\") pod \"authorino-58cd955675-7dl7s\" (UID: \"4465214e-083c-417c-b2c3-d9730ecfd403\") " pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:26.401429 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.401403 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:07:26.403761 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:07:26.403720 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0d9744_f573_4a22_bff5_ab017b94c7bb.slice/crio-ae95a578e3194628baac11ac1043dc5d40582f5cb94c1b740d1aca4074af1b6b WatchSource:0}: Error finding container ae95a578e3194628baac11ac1043dc5d40582f5cb94c1b740d1aca4074af1b6b: Status 404 returned error can't find the container with id ae95a578e3194628baac11ac1043dc5d40582f5cb94c1b740d1aca4074af1b6b Jul 09 04:07:26.485431 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.485403 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:07:26.521850 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.521810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:26.532921 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.532895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:07:26.533032 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.532982 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:07:26.557445 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.557413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28d8\" (UniqueName: \"kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8\") pod \"authorino-75d556bd77-vhrkz\" (UID: \"0b584072-312e-498f-ba1f-3658251cf62a\") " pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:07:26.658325 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.658296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28d8\" (UniqueName: \"kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8\") pod \"authorino-75d556bd77-vhrkz\" (UID: \"0b584072-312e-498f-ba1f-3658251cf62a\") " pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:07:26.677999 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.677964 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28d8\" (UniqueName: \"kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8\") pod \"authorino-75d556bd77-vhrkz\" (UID: \"0b584072-312e-498f-ba1f-3658251cf62a\") " pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:07:26.680198 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.680181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:26.681586 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:07:26.681559 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4465214e_083c_417c_b2c3_d9730ecfd403.slice/crio-66f4ce0ce1c8962d9ca4cb0f70768786ef80dcc9fd0c582ded4241e2aac4b75d WatchSource:0}: Error finding container 66f4ce0ce1c8962d9ca4cb0f70768786ef80dcc9fd0c582ded4241e2aac4b75d: Status 404 returned error can't find the container with id 66f4ce0ce1c8962d9ca4cb0f70768786ef80dcc9fd0c582ded4241e2aac4b75d Jul 09 04:07:26.842272 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:26.842199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:07:27.040891 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:27.040829 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:07:27.044840 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:07:27.044809 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b584072_312e_498f_ba1f_3658251cf62a.slice/crio-d0866876a1deabf3d7f31d716e9c375e6245dc221842ece369f76441a857226a WatchSource:0}: Error finding container d0866876a1deabf3d7f31d716e9c375e6245dc221842ece369f76441a857226a: Status 404 returned error can't find the container with id d0866876a1deabf3d7f31d716e9c375e6245dc221842ece369f76441a857226a Jul 09 04:07:27.319929 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:27.319887 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vhrkz" event={"ID":"0b584072-312e-498f-ba1f-3658251cf62a","Type":"ContainerStarted","Data":"d0866876a1deabf3d7f31d716e9c375e6245dc221842ece369f76441a857226a"} Jul 09 04:07:27.322312 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:27.322265 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7dl7s" event={"ID":"4465214e-083c-417c-b2c3-d9730ecfd403","Type":"ContainerStarted","Data":"66f4ce0ce1c8962d9ca4cb0f70768786ef80dcc9fd0c582ded4241e2aac4b75d"} Jul 09 04:07:27.323711 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:27.323676 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" event={"ID":"8c0d9744-f573-4a22-bff5-ab017b94c7bb","Type":"ContainerStarted","Data":"ae95a578e3194628baac11ac1043dc5d40582f5cb94c1b740d1aca4074af1b6b"} Jul 09 04:07:33.362609 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.362565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vhrkz" event={"ID":"0b584072-312e-498f-ba1f-3658251cf62a","Type":"ContainerStarted","Data":"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0"} Jul 09 04:07:33.363977 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.363953 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7dl7s" event={"ID":"4465214e-083c-417c-b2c3-d9730ecfd403","Type":"ContainerStarted","Data":"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962"} Jul 09 04:07:33.365398 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.365378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" event={"ID":"8c0d9744-f573-4a22-bff5-ab017b94c7bb","Type":"ContainerStarted","Data":"c37d1288c2125de9958c761edc8acc83c042e7c8fd94fc21edc94c77816fa5dd"} Jul 09 04:07:33.365509 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.365497 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:33.378918 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.378873 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vhrkz" podStartSLOduration=1.961567555 podStartE2EDuration="7.378859785s" totalImagesPullingTime="5.41729223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:33.37792847 +0000 UTC m=+439.493105508" watchObservedRunningTime="2026-07-09 04:07:33.378859785 +0000 UTC m=+439.494036884" Jul 09 04:07:33.397625 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.397575 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" podStartSLOduration=2.503301407 podStartE2EDuration="8.397561322s" totalImagesPullingTime="5.894259915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:33.396346523 +0000 UTC m=+439.511523560" watchObservedRunningTime="2026-07-09 04:07:33.397561322 +0000 UTC m=+439.512738350" Jul 09 04:07:33.409507 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.409467 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:33.415822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:33.415769 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-7dl7s" podStartSLOduration=1.622005184 podStartE2EDuration="7.415754453s" totalImagesPullingTime="5.793749269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:33.415537598 +0000 UTC m=+439.530714637" watchObservedRunningTime="2026-07-09 04:07:33.415754453 +0000 UTC m=+439.530931485" Jul 09 04:07:35.375583 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:35.375499 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-7dl7s" podUID="4465214e-083c-417c-b2c3-d9730ecfd403" containerName="authorino" containerID="cri-o://a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962" gracePeriod=30 Jul 09 04:07:35.610031 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:35.610008 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:35.740598 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:35.740562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k47zm\" (UniqueName: \"kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm\") pod \"4465214e-083c-417c-b2c3-d9730ecfd403\" (UID: \"4465214e-083c-417c-b2c3-d9730ecfd403\") " Jul 09 04:07:35.742743 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:35.742717 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm" (OuterVolumeSpecName: "kube-api-access-k47zm") pod "4465214e-083c-417c-b2c3-d9730ecfd403" (UID: "4465214e-083c-417c-b2c3-d9730ecfd403"). InnerVolumeSpecName "kube-api-access-k47zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:35.842059 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:35.842013 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47zm\" (UniqueName: \"kubernetes.io/projected/4465214e-083c-417c-b2c3-d9730ecfd403-kube-api-access-k47zm\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.122530 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.122433 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7859884ffc-jv76z" podUID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" containerName="console" containerID="cri-o://4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b" gracePeriod=15 Jul 09 04:07:36.348200 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.348172 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:07:36.381487 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.381403 2570 generic.go:356] "Generic (PLEG): container finished" podID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" containerID="4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b" exitCode=0 Jul 09 04:07:36.381487 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.381472 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7859884ffc-jv76z" Jul 09 04:07:36.381975 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.381480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7859884ffc-jv76z" event={"ID":"dfccc6e6-fa9d-4221-9512-cb9f856adee2","Type":"ContainerDied","Data":"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b"} Jul 09 04:07:36.381975 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.381593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7859884ffc-jv76z" event={"ID":"dfccc6e6-fa9d-4221-9512-cb9f856adee2","Type":"ContainerDied","Data":"7be9b1ffc95ba5ec655a645ffd9e9533f0b24029ebbfb2df2f7ea54c4f7d9fde"} Jul 09 04:07:36.381975 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.381622 2570 scope.go:117] "RemoveContainer" containerID="4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b" Jul 09 04:07:36.383085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.382665 2570 generic.go:356] "Generic (PLEG): container finished" podID="4465214e-083c-417c-b2c3-d9730ecfd403" containerID="a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962" exitCode=0 Jul 09 04:07:36.383085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.382705 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7dl7s" Jul 09 04:07:36.383085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.382802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7dl7s" event={"ID":"4465214e-083c-417c-b2c3-d9730ecfd403","Type":"ContainerDied","Data":"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962"} Jul 09 04:07:36.383085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.382839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7dl7s" event={"ID":"4465214e-083c-417c-b2c3-d9730ecfd403","Type":"ContainerDied","Data":"66f4ce0ce1c8962d9ca4cb0f70768786ef80dcc9fd0c582ded4241e2aac4b75d"} Jul 09 04:07:36.390148 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.390120 2570 scope.go:117] "RemoveContainer" containerID="4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b" Jul 09 04:07:36.390386 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:07:36.390369 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b\": container with ID starting with 4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b not found: ID does not exist" containerID="4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b" Jul 09 04:07:36.390449 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.390396 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b"} err="failed to get container status \"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b\": rpc error: code = NotFound desc = could not find container \"4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b\": container with ID starting with 4f6c5bda18e0350073e7f395ac4a8d39fb468aeb75a45d629aa9f9414a86557b not found: ID does not exist" Jul 09 04:07:36.390449 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.390412 2570 scope.go:117] "RemoveContainer" containerID="a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962" Jul 09 04:07:36.398091 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.398069 2570 scope.go:117] "RemoveContainer" containerID="a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962" Jul 09 04:07:36.398346 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:07:36.398327 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962\": container with ID starting with a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962 not found: ID does not exist" containerID="a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962" Jul 09 04:07:36.398408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.398353 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962"} err="failed to get container status \"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962\": rpc error: code = NotFound desc = could not find container \"a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962\": container with ID starting with a01d686c770ab1ed9041230e919c500ab115a9fb3b455dbfbe4da7743c742962 not found: ID does not exist" Jul 09 04:07:36.405277 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.405255 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:36.410987 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.410965 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-7dl7s"] Jul 09 04:07:36.427222 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.427198 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4465214e-083c-417c-b2c3-d9730ecfd403" path="/var/lib/kubelet/pods/4465214e-083c-417c-b2c3-d9730ecfd403/volumes" Jul 09 04:07:36.447116 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447085 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447229 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447142 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl2tw\" (UniqueName: \"kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447229 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447170 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447229 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447200 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447379 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447235 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447379 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447321 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447472 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447381 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert\") pod \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\" (UID: \"dfccc6e6-fa9d-4221-9512-cb9f856adee2\") " Jul 09 04:07:36.447538 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447518 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:07:36.447638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447606 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config" (OuterVolumeSpecName: "console-config") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:07:36.447813 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447758 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-trusted-ca-bundle\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.447866 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447816 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca" (OuterVolumeSpecName: "service-ca") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:07:36.447866 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.447839 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 04:07:36.449468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.449444 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:07:36.449569 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.449548 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw" (OuterVolumeSpecName: "kube-api-access-kl2tw") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "kube-api-access-kl2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:36.449608 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.449572 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dfccc6e6-fa9d-4221-9512-cb9f856adee2" (UID: "dfccc6e6-fa9d-4221-9512-cb9f856adee2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:07:36.549173 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549137 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2tw\" (UniqueName: \"kubernetes.io/projected/dfccc6e6-fa9d-4221-9512-cb9f856adee2-kube-api-access-kl2tw\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.549173 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549167 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-oauth-config\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.549173 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549178 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-config\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.549384 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549187 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfccc6e6-fa9d-4221-9512-cb9f856adee2-console-serving-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.549384 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549195 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-service-ca\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.549384 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.549203 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfccc6e6-fa9d-4221-9512-cb9f856adee2-oauth-serving-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:36.732055 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.732012 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:07:36.738012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:36.737989 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7859884ffc-jv76z"] Jul 09 04:07:38.426769 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:38.426735 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" path="/var/lib/kubelet/pods/dfccc6e6-fa9d-4221-9512-cb9f856adee2/volumes" Jul 09 04:07:44.373408 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:44.373379 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m5c6b" Jul 09 04:07:58.857931 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.857900 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ktlm9"] Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858200 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4465214e-083c-417c-b2c3-d9730ecfd403" containerName="authorino" Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858212 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4465214e-083c-417c-b2c3-d9730ecfd403" containerName="authorino" Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858226 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" containerName="console" Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858231 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" containerName="console" Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858276 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfccc6e6-fa9d-4221-9512-cb9f856adee2" containerName="console" Jul 09 04:07:58.858335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.858283 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4465214e-083c-417c-b2c3-d9730ecfd403" containerName="authorino" Jul 09 04:07:58.861628 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.861612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:58.867900 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:58.867879 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ktlm9"] Jul 09 04:07:59.038098 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.038060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgq6\" (UniqueName: \"kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6\") pod \"authorino-5d4b67c446-ktlm9\" (UID: \"7e03967d-be5b-43b3-84f2-dba038cc3df6\") " pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:59.050221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.050191 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ktlm9"] Jul 09 04:07:59.050419 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:07:59.050402 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4cgq6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-ktlm9" podUID="7e03967d-be5b-43b3-84f2-dba038cc3df6" Jul 09 04:07:59.078524 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.078498 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-ffgdq"] Jul 09 04:07:59.080780 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.080765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.084606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.084587 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 04:07:59.089638 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.089617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-ffgdq"] Jul 09 04:07:59.128729 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.128653 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-ffgdq"] Jul 09 04:07:59.128919 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:07:59.128899 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gdssb tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-gdssb tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-ffgdq" podUID="5a246c32-8112-423f-87a5-3c3faba4c88d" Jul 09 04:07:59.138981 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.138954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgq6\" (UniqueName: \"kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6\") pod \"authorino-5d4b67c446-ktlm9\" (UID: \"7e03967d-be5b-43b3-84f2-dba038cc3df6\") " pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:59.155657 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.155631 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgq6\" (UniqueName: \"kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6\") pod \"authorino-5d4b67c446-ktlm9\" (UID: \"7e03967d-be5b-43b3-84f2-dba038cc3df6\") " pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:59.161151 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.161127 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:07:59.163394 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.163379 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.173254 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.173235 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:07:59.239533 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.239504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.239679 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.239541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdssb\" (UniqueName: \"kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.340433 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.340402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdssb\" (UniqueName: \"kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.340584 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.340452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvf7\" (UniqueName: \"kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.340584 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.340490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.340584 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.340524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.342991 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.342968 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.348051 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.348019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdssb\" (UniqueName: \"kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb\") pod \"authorino-76877994-ffgdq\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.441574 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.441543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvf7\" (UniqueName: \"kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.441736 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.441597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.444231 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.444208 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.450903 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.450873 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvf7\" (UniqueName: \"kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7\") pod \"authorino-7c865f96d8-hwztx\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.472661 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.472641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:07:59.486261 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.486242 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.486355 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.486259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:59.493452 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.493431 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:07:59.501658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.501639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:07:59.618755 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.618732 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:07:59.620503 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:07:59.620480 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ce7041f_1175_476a_882c_05a0431145a3.slice/crio-d7de92597742b087fe929a872b83fd3cb017cff3ba8b2eabc0d15f814fbb4181 WatchSource:0}: Error finding container d7de92597742b087fe929a872b83fd3cb017cff3ba8b2eabc0d15f814fbb4181: Status 404 returned error can't find the container with id d7de92597742b087fe929a872b83fd3cb017cff3ba8b2eabc0d15f814fbb4181 Jul 09 04:07:59.646288 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.646262 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdssb\" (UniqueName: \"kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb\") pod \"5a246c32-8112-423f-87a5-3c3faba4c88d\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " Jul 09 04:07:59.646393 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.646326 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cgq6\" (UniqueName: \"kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6\") pod \"7e03967d-be5b-43b3-84f2-dba038cc3df6\" (UID: \"7e03967d-be5b-43b3-84f2-dba038cc3df6\") " Jul 09 04:07:59.646393 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.646384 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert\") pod \"5a246c32-8112-423f-87a5-3c3faba4c88d\" (UID: \"5a246c32-8112-423f-87a5-3c3faba4c88d\") " Jul 09 04:07:59.648520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.648491 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5a246c32-8112-423f-87a5-3c3faba4c88d" (UID: "5a246c32-8112-423f-87a5-3c3faba4c88d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:07:59.648520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.648504 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6" (OuterVolumeSpecName: "kube-api-access-4cgq6") pod "7e03967d-be5b-43b3-84f2-dba038cc3df6" (UID: "7e03967d-be5b-43b3-84f2-dba038cc3df6"). InnerVolumeSpecName "kube-api-access-4cgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:59.648647 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.648518 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb" (OuterVolumeSpecName: "kube-api-access-gdssb") pod "5a246c32-8112-423f-87a5-3c3faba4c88d" (UID: "5a246c32-8112-423f-87a5-3c3faba4c88d"). InnerVolumeSpecName "kube-api-access-gdssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:07:59.747941 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.747862 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cgq6\" (UniqueName: \"kubernetes.io/projected/7e03967d-be5b-43b3-84f2-dba038cc3df6-kube-api-access-4cgq6\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:59.747941 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.747890 2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a246c32-8112-423f-87a5-3c3faba4c88d-tls-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:07:59.747941 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:07:59.747902 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdssb\" (UniqueName: \"kubernetes.io/projected/5a246c32-8112-423f-87a5-3c3faba4c88d-kube-api-access-gdssb\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:08:00.492916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.492875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c865f96d8-hwztx" event={"ID":"9ce7041f-1175-476a-882c-05a0431145a3","Type":"ContainerStarted","Data":"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063"} Jul 09 04:08:00.492916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.492897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ktlm9" Jul 09 04:08:00.493349 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.492923 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c865f96d8-hwztx" event={"ID":"9ce7041f-1175-476a-882c-05a0431145a3","Type":"ContainerStarted","Data":"d7de92597742b087fe929a872b83fd3cb017cff3ba8b2eabc0d15f814fbb4181"} Jul 09 04:08:00.493349 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.492928 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ffgdq" Jul 09 04:08:00.518437 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.518399 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ktlm9"] Jul 09 04:08:00.520117 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.520095 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ktlm9"] Jul 09 04:08:00.531861 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.531768 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c865f96d8-hwztx" podStartSLOduration=1.199023261 podStartE2EDuration="1.531752308s" totalImagesPullingTime="332.729047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:00.53175138 +0000 UTC m=+466.646928420" watchObservedRunningTime="2026-07-09 04:08:00.531752308 +0000 UTC m=+466.646929347" Jul 09 04:08:00.554670 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.554641 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-ffgdq"] Jul 09 04:08:00.558909 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.558884 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:08:00.559086 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.559063 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vhrkz" podUID="0b584072-312e-498f-ba1f-3658251cf62a" containerName="authorino" containerID="cri-o://e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0" gracePeriod=30 Jul 09 04:08:00.561624 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.561601 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-ffgdq"] Jul 09 04:08:00.829672 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.829645 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:08:00.958198 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.958171 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d28d8\" (UniqueName: \"kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8\") pod \"0b584072-312e-498f-ba1f-3658251cf62a\" (UID: \"0b584072-312e-498f-ba1f-3658251cf62a\") " Jul 09 04:08:00.960319 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:00.960289 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8" (OuterVolumeSpecName: "kube-api-access-d28d8") pod "0b584072-312e-498f-ba1f-3658251cf62a" (UID: "0b584072-312e-498f-ba1f-3658251cf62a"). InnerVolumeSpecName "kube-api-access-d28d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:08:01.058895 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.058828 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d28d8\" (UniqueName: \"kubernetes.io/projected/0b584072-312e-498f-ba1f-3658251cf62a-kube-api-access-d28d8\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:08:01.499025 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.498990 2570 generic.go:356] "Generic (PLEG): container finished" podID="0b584072-312e-498f-ba1f-3658251cf62a" containerID="e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0" exitCode=0 Jul 09 04:08:01.499468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.499031 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vhrkz" Jul 09 04:08:01.499468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.499084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vhrkz" event={"ID":"0b584072-312e-498f-ba1f-3658251cf62a","Type":"ContainerDied","Data":"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0"} Jul 09 04:08:01.499468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.499124 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vhrkz" event={"ID":"0b584072-312e-498f-ba1f-3658251cf62a","Type":"ContainerDied","Data":"d0866876a1deabf3d7f31d716e9c375e6245dc221842ece369f76441a857226a"} Jul 09 04:08:01.499468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.499145 2570 scope.go:117] "RemoveContainer" containerID="e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0" Jul 09 04:08:01.508614 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.508596 2570 scope.go:117] "RemoveContainer" containerID="e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0" Jul 09 04:08:01.508874 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:08:01.508855 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0\": container with ID starting with e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0 not found: ID does not exist" containerID="e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0" Jul 09 04:08:01.508923 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.508881 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0"} err="failed to get container status \"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0\": rpc error: code = NotFound desc = could not find container \"e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0\": container with ID starting with e65f468a017bc5b99d13ecf474fa7689917f661878171488fa40064ea5c7acd0 not found: ID does not exist" Jul 09 04:08:01.518744 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.518720 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:08:01.522166 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:01.522145 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vhrkz"] Jul 09 04:08:02.427211 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:02.427182 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b584072-312e-498f-ba1f-3658251cf62a" path="/var/lib/kubelet/pods/0b584072-312e-498f-ba1f-3658251cf62a/volumes" Jul 09 04:08:02.427542 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:02.427530 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a246c32-8112-423f-87a5-3c3faba4c88d" path="/var/lib/kubelet/pods/5a246c32-8112-423f-87a5-3c3faba4c88d/volumes" Jul 09 04:08:02.427713 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:02.427702 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e03967d-be5b-43b3-84f2-dba038cc3df6" path="/var/lib/kubelet/pods/7e03967d-be5b-43b3-84f2-dba038cc3df6/volumes" Jul 09 04:08:41.432945 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:08:41.432914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:05.538726 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:05.538643 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:11.740840 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:11.740156 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:27.151654 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:27.151619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:38.735096 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:38.735036 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:50.536859 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:50.536818 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:09:57.829037 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:09:57.828999 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:10:08.533394 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.533362 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk"] Jul 09 04:10:08.533843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.533810 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b584072-312e-498f-ba1f-3658251cf62a" containerName="authorino" Jul 09 04:10:08.533843 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.533827 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b584072-312e-498f-ba1f-3658251cf62a" containerName="authorino" Jul 09 04:10:08.533955 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.533902 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b584072-312e-498f-ba1f-3658251cf62a" containerName="authorino" Jul 09 04:10:08.536778 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.536758 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.540556 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.540530 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j7ms6\"" Jul 09 04:10:08.540820 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.540549 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 04:10:08.540820 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.540564 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 04:10:08.540820 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.540564 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 09 04:10:08.546879 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.546859 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk"] Jul 09 04:10:08.699661 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699626 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.699661 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjb9\" (UniqueName: \"kubernetes.io/projected/a064056d-884b-4cb6-b362-854a88a55ba9-kube-api-access-bbjb9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.699874 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.699874 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.699874 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a064056d-884b-4cb6-b362-854a88a55ba9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.699874 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.699813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800328 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800328 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800328 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjb9\" (UniqueName: \"kubernetes.io/projected/a064056d-884b-4cb6-b362-854a88a55ba9-kube-api-access-bbjb9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800328 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800668 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800668 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a064056d-884b-4cb6-b362-854a88a55ba9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800764 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800718 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.800838 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.800773 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.802733 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.802706 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a064056d-884b-4cb6-b362-854a88a55ba9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.802966 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.802949 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a064056d-884b-4cb6-b362-854a88a55ba9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.807891 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.807873 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjb9\" (UniqueName: \"kubernetes.io/projected/a064056d-884b-4cb6-b362-854a88a55ba9-kube-api-access-bbjb9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk\" (UID: \"a064056d-884b-4cb6-b362-854a88a55ba9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.846666 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.846640 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:08.980529 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:08.980503 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk"] Jul 09 04:10:08.982665 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:10:08.982643 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda064056d_884b_4cb6_b362_854a88a55ba9.slice/crio-46839b5fd864fd90115ff79f575cf345c2911be56d2a1ce79009f6709dfcc8df WatchSource:0}: Error finding container 46839b5fd864fd90115ff79f575cf345c2911be56d2a1ce79009f6709dfcc8df: Status 404 returned error can't find the container with id 46839b5fd864fd90115ff79f575cf345c2911be56d2a1ce79009f6709dfcc8df Jul 09 04:10:09.074667 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:09.074574 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" event={"ID":"a064056d-884b-4cb6-b362-854a88a55ba9","Type":"ContainerStarted","Data":"46839b5fd864fd90115ff79f575cf345c2911be56d2a1ce79009f6709dfcc8df"} Jul 09 04:10:10.837335 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:10.837297 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:10:11.091480 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:11.091385 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" event={"ID":"a064056d-884b-4cb6-b362-854a88a55ba9","Type":"ContainerStarted","Data":"6f5bf8d3a879644cc65814cbbac47795e6785af1a8ca9d9b9aa6db7cd54d95df"} Jul 09 04:10:11.107919 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:11.107868 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" podStartSLOduration=1.996860799 podStartE2EDuration="3.10784946s" totalImagesPullingTime="1.110988661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:10:11.107197203 +0000 UTC m=+597.222374265" watchObservedRunningTime="2026-07-09 04:10:11.10784946 +0000 UTC m=+597.223026497" Jul 09 04:10:14.323606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:14.323579 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:10:14.324033 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:14.324004 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:10:18.847560 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:18.847513 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:18.847926 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:18.847578 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:18.860164 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:18.860135 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:10:19.138207 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:10:19.138124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk" Jul 09 04:11:20.343621 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.343582 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bbc96db9-mpdf4"] Jul 09 04:11:20.346726 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.346706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.353991 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.353896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bbc96db9-mpdf4"] Jul 09 04:11:20.495345 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.495310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3763028c-44e2-4ab3-8b3e-7af799e783c5-tls-cert\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.495537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.495499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbgc\" (UniqueName: \"kubernetes.io/projected/3763028c-44e2-4ab3-8b3e-7af799e783c5-kube-api-access-mxbgc\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.596213 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.596136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3763028c-44e2-4ab3-8b3e-7af799e783c5-tls-cert\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.596213 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.596201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbgc\" (UniqueName: \"kubernetes.io/projected/3763028c-44e2-4ab3-8b3e-7af799e783c5-kube-api-access-mxbgc\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.598839 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.598814 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3763028c-44e2-4ab3-8b3e-7af799e783c5-tls-cert\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.603537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.603512 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbgc\" (UniqueName: \"kubernetes.io/projected/3763028c-44e2-4ab3-8b3e-7af799e783c5-kube-api-access-mxbgc\") pod \"authorino-7bbc96db9-mpdf4\" (UID: \"3763028c-44e2-4ab3-8b3e-7af799e783c5\") " pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.656351 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.656319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bbc96db9-mpdf4" Jul 09 04:11:20.785494 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.785440 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bbc96db9-mpdf4"] Jul 09 04:11:20.791022 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:20.790994 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:11:21.423267 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.423226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bbc96db9-mpdf4" event={"ID":"3763028c-44e2-4ab3-8b3e-7af799e783c5","Type":"ContainerStarted","Data":"cae0a0c09e3cb0342ddb8c0b4fc65fe70b451d5cb15f0bda7575073d0844c744"} Jul 09 04:11:21.423267 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.423271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bbc96db9-mpdf4" event={"ID":"3763028c-44e2-4ab3-8b3e-7af799e783c5","Type":"ContainerStarted","Data":"341fb4ae80b00020b2922e2c0403237a578898a9268681b3ab8dc2943d0c88d8"} Jul 09 04:11:21.457707 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.457648 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bbc96db9-mpdf4" podStartSLOduration=1.107965072 podStartE2EDuration="1.457630096s" totalImagesPullingTime="349.665024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:11:21.455889437 +0000 UTC m=+667.571066474" watchObservedRunningTime="2026-07-09 04:11:21.457630096 +0000 UTC m=+667.572807131" Jul 09 04:11:21.528337 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.528247 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:11:21.528768 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.528502 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7c865f96d8-hwztx" podUID="9ce7041f-1175-476a-882c-05a0431145a3" containerName="authorino" containerID="cri-o://1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063" gracePeriod=30 Jul 09 04:11:21.787093 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.787066 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:11:21.908639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.908607 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert\") pod \"9ce7041f-1175-476a-882c-05a0431145a3\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " Jul 09 04:11:21.908838 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.908725 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgvf7\" (UniqueName: \"kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7\") pod \"9ce7041f-1175-476a-882c-05a0431145a3\" (UID: \"9ce7041f-1175-476a-882c-05a0431145a3\") " Jul 09 04:11:21.911496 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.911456 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7" (OuterVolumeSpecName: "kube-api-access-wgvf7") pod "9ce7041f-1175-476a-882c-05a0431145a3" (UID: "9ce7041f-1175-476a-882c-05a0431145a3"). InnerVolumeSpecName "kube-api-access-wgvf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:11:21.918906 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:21.918881 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9ce7041f-1175-476a-882c-05a0431145a3" (UID: "9ce7041f-1175-476a-882c-05a0431145a3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:11:22.010323 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.010233 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvf7\" (UniqueName: \"kubernetes.io/projected/9ce7041f-1175-476a-882c-05a0431145a3-kube-api-access-wgvf7\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:11:22.010323 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.010262 2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ce7041f-1175-476a-882c-05a0431145a3-tls-cert\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:11:22.430213 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.430179 2570 generic.go:356] "Generic (PLEG): container finished" podID="9ce7041f-1175-476a-882c-05a0431145a3" containerID="1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063" exitCode=0 Jul 09 04:11:22.430659 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.430221 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c865f96d8-hwztx" Jul 09 04:11:22.430659 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.430267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c865f96d8-hwztx" event={"ID":"9ce7041f-1175-476a-882c-05a0431145a3","Type":"ContainerDied","Data":"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063"} Jul 09 04:11:22.430659 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.430308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c865f96d8-hwztx" event={"ID":"9ce7041f-1175-476a-882c-05a0431145a3","Type":"ContainerDied","Data":"d7de92597742b087fe929a872b83fd3cb017cff3ba8b2eabc0d15f814fbb4181"} Jul 09 04:11:22.430659 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.430329 2570 scope.go:117] "RemoveContainer" containerID="1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063" Jul 09 04:11:22.447754 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.447735 2570 scope.go:117] "RemoveContainer" containerID="1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063" Jul 09 04:11:22.448094 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:11:22.448074 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063\": container with ID starting with 1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063 not found: ID does not exist" containerID="1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063" Jul 09 04:11:22.448170 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.448103 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063"} err="failed to get container status \"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063\": rpc error: code = NotFound desc = could not find container \"1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063\": container with ID starting with 1ea197df1f1fb5b1507bfe2ed6024b92cbbaa68bc5d880d1919487eeae161063 not found: ID does not exist" Jul 09 04:11:22.460402 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.460376 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:11:22.464768 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:22.464746 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c865f96d8-hwztx"] Jul 09 04:11:24.426943 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:24.426908 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce7041f-1175-476a-882c-05a0431145a3" path="/var/lib/kubelet/pods/9ce7041f-1175-476a-882c-05a0431145a3/volumes" Jul 09 04:11:50.740237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:11:50.740201 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:12:00.531390 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:00.531357 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:12:11.931219 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:11.931186 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:12:20.441462 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:20.441427 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:12:31.629097 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:31.629062 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:12:34.705012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.704975 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-z26kh"] Jul 09 04:12:34.705390 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.705315 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce7041f-1175-476a-882c-05a0431145a3" containerName="authorino" Jul 09 04:12:34.705390 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.705327 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce7041f-1175-476a-882c-05a0431145a3" containerName="authorino" Jul 09 04:12:34.705390 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.705379 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce7041f-1175-476a-882c-05a0431145a3" containerName="authorino" Jul 09 04:12:34.708967 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.708950 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.714243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.714211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 04:12:34.714392 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.714213 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7z8mm\"" Jul 09 04:12:34.714392 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.714244 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 04:12:34.721998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.721974 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-z26kh"] Jul 09 04:12:34.833782 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.833742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sd2r\" (UniqueName: \"kubernetes.io/projected/f06f6af2-5610-4147-8b3e-63261b243bce-kube-api-access-2sd2r\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.833962 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.833824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f06f6af2-5610-4147-8b3e-63261b243bce-webhook-cert\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.934750 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.934714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f06f6af2-5610-4147-8b3e-63261b243bce-webhook-cert\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.934908 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.934767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sd2r\" (UniqueName: \"kubernetes.io/projected/f06f6af2-5610-4147-8b3e-63261b243bce-kube-api-access-2sd2r\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.937454 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.937432 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f06f6af2-5610-4147-8b3e-63261b243bce-webhook-cert\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:34.943756 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:34.943729 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sd2r\" (UniqueName: \"kubernetes.io/projected/f06f6af2-5610-4147-8b3e-63261b243bce-kube-api-access-2sd2r\") pod \"maas-controller-6cbfc7556c-z26kh\" (UID: \"f06f6af2-5610-4147-8b3e-63261b243bce\") " pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:35.019128 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:35.019006 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:35.148926 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:35.148887 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbfc7556c-z26kh"] Jul 09 04:12:35.151240 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:12:35.151210 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06f6af2_5610_4147_8b3e_63261b243bce.slice/crio-b4666ecbbb698c87c9639ac39a67cac9d807e84c5ca7c9a1b690a51d8142b80a WatchSource:0}: Error finding container b4666ecbbb698c87c9639ac39a67cac9d807e84c5ca7c9a1b690a51d8142b80a: Status 404 returned error can't find the container with id b4666ecbbb698c87c9639ac39a67cac9d807e84c5ca7c9a1b690a51d8142b80a Jul 09 04:12:35.770805 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:35.770769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" event={"ID":"f06f6af2-5610-4147-8b3e-63261b243bce","Type":"ContainerStarted","Data":"b4666ecbbb698c87c9639ac39a67cac9d807e84c5ca7c9a1b690a51d8142b80a"} Jul 09 04:12:37.782256 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:37.782219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" event={"ID":"f06f6af2-5610-4147-8b3e-63261b243bce","Type":"ContainerStarted","Data":"b852af59c50d317da5351188c2787daeeff325a677af5cda2c829c0472b0ae89"} Jul 09 04:12:37.782613 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:37.782369 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:12:37.801153 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:37.801105 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" podStartSLOduration=1.903283546 podStartE2EDuration="3.801090133s" totalImagesPullingTime="1.897806587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:12:37.798581207 +0000 UTC m=+743.913758243" watchObservedRunningTime="2026-07-09 04:12:37.801090133 +0000 UTC m=+743.916267169" Jul 09 04:12:48.791845 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:12:48.791818 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbfc7556c-z26kh" Jul 09 04:13:00.940202 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:13:00.940157 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:13:14.834365 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:13:14.834326 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:13:47.151232 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:13:47.151160 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:13:56.634998 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:13:56.634960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:07.336033 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:07.335993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:15.141171 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:15.141140 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:26.935823 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:26.935791 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:36.936842 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:36.936802 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:47.638498 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:47.638460 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:14:58.133978 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:14:58.133942 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:15:00.145153 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.145117 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:15:00.148567 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.148550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:15:00.151210 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.151185 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:15:00.152350 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.152329 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:15:00.152449 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.152348 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cfhx4\"" Jul 09 04:15:00.165061 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.165023 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:15:00.186995 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.186952 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdjd\" (UniqueName: \"kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd\") pod \"maas-api-key-cleanup-29726175-45z9d\" (UID: \"fcb27243-8142-4366-ae5b-f9aab72857ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:15:00.287934 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.287902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdjd\" (UniqueName: \"kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd\") pod \"maas-api-key-cleanup-29726175-45z9d\" (UID: \"fcb27243-8142-4366-ae5b-f9aab72857ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:15:00.296708 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.296685 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdjd\" (UniqueName: \"kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd\") pod \"maas-api-key-cleanup-29726175-45z9d\" (UID: \"fcb27243-8142-4366-ae5b-f9aab72857ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:15:00.480557 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.480515 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:15:00.608909 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:00.608884 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:15:00.610609 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:15:00.610586 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb27243_8142_4366_ae5b_f9aab72857ca.slice/crio-6883ca7a9309af556715c81d389f3d67efa3454ec00697d9d0193bc5c006c8db WatchSource:0}: Error finding container 6883ca7a9309af556715c81d389f3d67efa3454ec00697d9d0193bc5c006c8db: Status 404 returned error can't find the container with id 6883ca7a9309af556715c81d389f3d67efa3454ec00697d9d0193bc5c006c8db Jul 09 04:15:01.458551 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:01.458504 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" event={"ID":"fcb27243-8142-4366-ae5b-f9aab72857ca","Type":"ContainerStarted","Data":"6883ca7a9309af556715c81d389f3d67efa3454ec00697d9d0193bc5c006c8db"} Jul 09 04:15:04.478122 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:04.478083 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" event={"ID":"fcb27243-8142-4366-ae5b-f9aab72857ca","Type":"ContainerStarted","Data":"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9"} Jul 09 04:15:04.494234 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:04.494189 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" podStartSLOduration=0.974545802 podStartE2EDuration="4.494174636s" totalImagesPullingTime="3.519628834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:15:04.492871586 +0000 UTC m=+890.608048623" watchObservedRunningTime="2026-07-09 04:15:04.494174636 +0000 UTC m=+890.609351671" Jul 09 04:15:14.353501 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:14.353466 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:15:14.353926 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:14.353524 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:15:19.734017 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:19.733978 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:15:31.830004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:31.829964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:15:54.842311 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:15:54.842272 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:16:00.011804 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:00.011769 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:16:00.012211 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:00.011980 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" podUID="fcb27243-8142-4366-ae5b-f9aab72857ca" containerName="cleanup" containerID="cri-o://2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9" gracePeriod=30 Jul 09 04:16:05.336703 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:05.336666 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:16:20.841614 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:20.841568 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:16:30.184245 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.184219 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:16:30.231905 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.231874 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrdjd\" (UniqueName: \"kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd\") pod \"fcb27243-8142-4366-ae5b-f9aab72857ca\" (UID: \"fcb27243-8142-4366-ae5b-f9aab72857ca\") " Jul 09 04:16:30.234221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.234190 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd" (OuterVolumeSpecName: "kube-api-access-hrdjd") pod "fcb27243-8142-4366-ae5b-f9aab72857ca" (UID: "fcb27243-8142-4366-ae5b-f9aab72857ca"). InnerVolumeSpecName "kube-api-access-hrdjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:16:30.332691 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.332612 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrdjd\" (UniqueName: \"kubernetes.io/projected/fcb27243-8142-4366-ae5b-f9aab72857ca-kube-api-access-hrdjd\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:16:30.889727 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.889689 2570 generic.go:356] "Generic (PLEG): container finished" podID="fcb27243-8142-4366-ae5b-f9aab72857ca" containerID="2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9" exitCode=137 Jul 09 04:16:30.889916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.889748 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" Jul 09 04:16:30.889916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.889778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" event={"ID":"fcb27243-8142-4366-ae5b-f9aab72857ca","Type":"ContainerDied","Data":"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9"} Jul 09 04:16:30.889916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.889857 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d" event={"ID":"fcb27243-8142-4366-ae5b-f9aab72857ca","Type":"ContainerDied","Data":"6883ca7a9309af556715c81d389f3d67efa3454ec00697d9d0193bc5c006c8db"} Jul 09 04:16:30.889916 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.889879 2570 scope.go:117] "RemoveContainer" containerID="2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9" Jul 09 04:16:30.899541 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.899518 2570 scope.go:117] "RemoveContainer" containerID="2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9" Jul 09 04:16:30.899802 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:16:30.899785 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9\": container with ID starting with 2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9 not found: ID does not exist" containerID="2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9" Jul 09 04:16:30.899866 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.899811 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9"} err="failed to get container status \"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9\": rpc error: code = NotFound desc = could not find container \"2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9\": container with ID starting with 2d192df0939ee7899296c1d4fbc137859d8f340ea36622d7c0e64d4169265ca9 not found: ID does not exist" Jul 09 04:16:30.909087 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.909050 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:16:30.912244 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:30.912226 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726175-45z9d"] Jul 09 04:16:31.733209 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:31.733178 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:16:32.432885 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:32.432851 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb27243-8142-4366-ae5b-f9aab72857ca" path="/var/lib/kubelet/pods/fcb27243-8142-4366-ae5b-f9aab72857ca/volumes" Jul 09 04:16:41.338135 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:41.338003 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:16:51.138827 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:16:51.138785 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:17:00.433719 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:17:00.433684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:18:48.136623 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:18:48.136587 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:18:59.543376 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:18:59.543331 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:19:10.532520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:19:10.532476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:19:20.733499 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:19:20.733461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:19:32.234688 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:19:32.234641 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:19:42.933986 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:19:42.933908 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:19:51.635265 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:19:51.635228 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:20:02.432410 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:02.432378 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:20:13.236861 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:13.236830 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:20:14.382474 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:14.382443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:20:14.383149 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:14.383129 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:20:23.431299 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:23.431268 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:20:33.842181 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:33.842146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:20:58.040158 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:20:58.040118 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:05.440525 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:05.440432 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:15.332483 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:15.332448 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:25.040575 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:25.040532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:31.625707 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:31.625675 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:47.047538 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:47.047509 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:21:58.230669 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:21:58.230632 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:07.924277 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:07.924228 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:18.539929 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:18.539890 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:27.837909 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:27.837870 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:36.730367 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:36.730271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:50.035576 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:50.035507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:22:59.136150 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:22:59.136113 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:23:08.530019 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:23:08.529979 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:23:19.428166 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:23:19.428131 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:23:30.638087 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:23:30.638054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:23:37.143427 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:23:37.143391 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:23:53.646828 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:23:53.645109 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:24:06.440473 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:24:06.440393 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:24:17.629745 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:24:17.629708 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:24:29.930746 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:24:29.930702 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:24:39.437495 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:24:39.437454 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:24:49.046999 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:24:49.046960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:25:06.131799 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:25:06.131761 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:25:14.419100 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:25:14.419074 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:25:14.420708 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:25:14.420688 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:25:16.938243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:25:16.938213 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:26:54.052185 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:26:54.052149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:27:03.026208 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:27:03.026175 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:27:15.853168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:27:15.853134 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:27:27.333872 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:27:27.333836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:27:55.434585 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:27:55.434541 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:28:09.439793 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.439759 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:09.440317 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.440269 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb27243-8142-4366-ae5b-f9aab72857ca" containerName="cleanup" Jul 09 04:28:09.440317 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.440290 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb27243-8142-4366-ae5b-f9aab72857ca" containerName="cleanup" Jul 09 04:28:09.440434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.440369 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb27243-8142-4366-ae5b-f9aab72857ca" containerName="cleanup" Jul 09 04:28:09.443309 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.443288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.447514 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.447482 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11e0a3fa\"/\"default-dockercfg-lx6mn\"" Jul 09 04:28:09.447633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.447511 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11e0a3fa\"/\"kube-root-ca.crt\"" Jul 09 04:28:09.447633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.447482 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11e0a3fa\"/\"e2e-lifecycle-model-11e0a3fa-kserve-self-signed-certs\"" Jul 09 04:28:09.447633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.447541 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11e0a3fa\"/\"openshift-service-ca.crt\"" Jul 09 04:28:09.452305 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.452281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:09.498108 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.498262 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.498262 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.498382 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498269 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.498382 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.498382 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.498304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhn6\" (UniqueName: \"kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599206 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599206 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599457 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599457 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599457 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599457 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhn6\" (UniqueName: \"kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599682 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599704 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599693 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.599788 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.599755 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.601789 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.601756 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.602002 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.601983 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.610294 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.610262 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhn6\" (UniqueName: \"kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6\") pod \"e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.752693 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.752605 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:09.889013 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:09.888990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:09.890648 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:28:09.890622 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd55ac992_764a_432b_b824_6c2baab50769.slice/crio-a9b62de8dd96aad5651b234859d087c7d5dc983869b494c06c75bf5ed7cedc8c WatchSource:0}: Error finding container a9b62de8dd96aad5651b234859d087c7d5dc983869b494c06c75bf5ed7cedc8c: Status 404 returned error can't find the container with id a9b62de8dd96aad5651b234859d087c7d5dc983869b494c06c75bf5ed7cedc8c Jul 09 04:28:10.131479 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:10.131396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" event={"ID":"d55ac992-764a-432b-b824-6c2baab50769","Type":"ContainerStarted","Data":"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a"} Jul 09 04:28:10.131479 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:10.131434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" event={"ID":"d55ac992-764a-432b-b824-6c2baab50769","Type":"ContainerStarted","Data":"a9b62de8dd96aad5651b234859d087c7d5dc983869b494c06c75bf5ed7cedc8c"} Jul 09 04:28:10.148742 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:10.148695 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" podStartSLOduration=1.148681584 podStartE2EDuration="1.148681584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:28:10.147439795 +0000 UTC m=+1676.262616831" watchObservedRunningTime="2026-07-09 04:28:10.148681584 +0000 UTC m=+1676.263858619" Jul 09 04:28:12.733607 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:12.733570 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:28:19.752899 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:19.752862 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:19.753324 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:19.752914 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:19.765736 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:19.765709 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:20.187124 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:20.187092 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:26.639337 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:26.639304 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:28:39.981627 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:39.981518 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:40.127238 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:40.127203 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:28:55.002592 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.002521 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" podUID="d55ac992-764a-432b-b824-6c2baab50769" containerName="main" containerID="cri-o://d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a" gracePeriod=45 Jul 09 04:28:55.242031 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.242003 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:55.293724 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.293648 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.293724 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.293685 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.293925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.293747 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.293925 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.293900 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache" (OuterVolumeSpecName: "model-cache") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:55.294029 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.293996 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-model-cache\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.294126 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.294101 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:55.295963 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.295943 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:28:55.338505 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.338472 2570 generic.go:356] "Generic (PLEG): container finished" podID="d55ac992-764a-432b-b824-6c2baab50769" containerID="d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a" exitCode=0 Jul 09 04:28:55.338678 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.338515 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" Jul 09 04:28:55.338678 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.338533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" event={"ID":"d55ac992-764a-432b-b824-6c2baab50769","Type":"ContainerDied","Data":"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a"} Jul 09 04:28:55.338678 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.338565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c" event={"ID":"d55ac992-764a-432b-b824-6c2baab50769","Type":"ContainerDied","Data":"a9b62de8dd96aad5651b234859d087c7d5dc983869b494c06c75bf5ed7cedc8c"} Jul 09 04:28:55.338678 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.338582 2570 scope.go:117] "RemoveContainer" containerID="d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a" Jul 09 04:28:55.349136 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.349115 2570 scope.go:117] "RemoveContainer" containerID="d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a" Jul 09 04:28:55.349428 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:28:55.349408 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a\": container with ID starting with d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a not found: ID does not exist" containerID="d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a" Jul 09 04:28:55.349486 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.349439 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a"} err="failed to get container status \"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a\": rpc error: code = NotFound desc = could not find container \"d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a\": container with ID starting with d3bacf4a141cec8399903a2f9bb8e8cb46361c9d9b09366680bf58fc261b7b8a not found: ID does not exist" Jul 09 04:28:55.394319 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394285 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzhn6\" (UniqueName: \"kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.394491 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394353 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.394491 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394402 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm\") pod \"d55ac992-764a-432b-b824-6c2baab50769\" (UID: \"d55ac992-764a-432b-b824-6c2baab50769\") " Jul 09 04:28:55.394620 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394597 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home" (OuterVolumeSpecName: "home") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:55.394730 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394662 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-tmp-dir\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.394730 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.394681 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55ac992-764a-432b-b824-6c2baab50769-tls-certs\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.396594 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.396568 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm" (OuterVolumeSpecName: "dshm") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:28:55.396695 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.396588 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6" (OuterVolumeSpecName: "kube-api-access-tzhn6") pod "d55ac992-764a-432b-b824-6c2baab50769" (UID: "d55ac992-764a-432b-b824-6c2baab50769"). InnerVolumeSpecName "kube-api-access-tzhn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:28:55.495329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.495293 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-home\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.495329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.495321 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55ac992-764a-432b-b824-6c2baab50769-dshm\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.495329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.495330 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhn6\" (UniqueName: \"kubernetes.io/projected/d55ac992-764a-432b-b824-6c2baab50769-kube-api-access-tzhn6\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:28:55.663489 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.663455 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:55.665131 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:55.665109 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-11e0a3fa/e2e-lifecycle-model-11e0a3fa-kserve-c9fb5f879-wxd7c"] Jul 09 04:28:56.427619 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:28:56.427583 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55ac992-764a-432b-b824-6c2baab50769" path="/var/lib/kubelet/pods/d55ac992-764a-432b-b824-6c2baab50769/volumes" Jul 09 04:29:19.636492 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:29:19.636455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:30:00.142664 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.142626 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:30:00.143096 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.142918 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55ac992-764a-432b-b824-6c2baab50769" containerName="main" Jul 09 04:30:00.143096 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.142928 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55ac992-764a-432b-b824-6c2baab50769" containerName="main" Jul 09 04:30:00.143096 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.142988 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55ac992-764a-432b-b824-6c2baab50769" containerName="main" Jul 09 04:30:00.145883 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.145867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:30:00.148639 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.148612 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 04:30:00.148773 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.148723 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cfhx4\"" Jul 09 04:30:00.149953 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.149933 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 04:30:00.152582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.152559 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:30:00.232391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.232358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv687\" (UniqueName: \"kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687\") pod \"maas-api-key-cleanup-29726190-5jdfv\" (UID: \"8bf9e2f3-7e78-411c-9334-729e861ee104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:30:00.332990 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.332955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv687\" (UniqueName: \"kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687\") pod \"maas-api-key-cleanup-29726190-5jdfv\" (UID: \"8bf9e2f3-7e78-411c-9334-729e861ee104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:30:00.341805 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.341773 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv687\" (UniqueName: \"kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687\") pod \"maas-api-key-cleanup-29726190-5jdfv\" (UID: \"8bf9e2f3-7e78-411c-9334-729e861ee104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:30:00.457481 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.457444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:30:00.590980 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.590948 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:30:00.592342 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:30:00.592317 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf9e2f3_7e78_411c_9334_729e861ee104.slice/crio-ce6e8bc9c0baf66aa2d4a8996a29faf26c1aa0fde240f42d908c295c4e62f683 WatchSource:0}: Error finding container ce6e8bc9c0baf66aa2d4a8996a29faf26c1aa0fde240f42d908c295c4e62f683: Status 404 returned error can't find the container with id ce6e8bc9c0baf66aa2d4a8996a29faf26c1aa0fde240f42d908c295c4e62f683 Jul 09 04:30:00.641120 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:00.641088 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" event={"ID":"8bf9e2f3-7e78-411c-9334-729e861ee104","Type":"ContainerStarted","Data":"ce6e8bc9c0baf66aa2d4a8996a29faf26c1aa0fde240f42d908c295c4e62f683"} Jul 09 04:30:01.646841 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:01.646802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" event={"ID":"8bf9e2f3-7e78-411c-9334-729e861ee104","Type":"ContainerStarted","Data":"87ac7f7c7195b0009903dc9e70270e59e36089845dd3399a3aae343b9007be8d"} Jul 09 04:30:01.663355 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:01.663303 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" podStartSLOduration=1.663287288 podStartE2EDuration="1.663287288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:01.660700016 +0000 UTC m=+1787.775877052" watchObservedRunningTime="2026-07-09 04:30:01.663287288 +0000 UTC m=+1787.778464325" Jul 09 04:30:14.449478 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:14.449444 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:30:14.451086 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:14.451069 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:30:27.735349 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:27.735319 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:30:38.838955 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:38.838921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:30:55.927091 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.927056 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:30:55.930601 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.930583 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:55.933242 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.933219 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7aa53b\"/\"kube-root-ca.crt\"" Jul 09 04:30:55.933355 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.933269 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7aa53b\"/\"openshift-service-ca.crt\"" Jul 09 04:30:55.934586 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.934570 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7aa53b\"/\"test-model-7aa53b-kserve-self-signed-certs\"" Jul 09 04:30:55.934673 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.934571 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7aa53b\"/\"default-dockercfg-rvvmp\"" Jul 09 04:30:55.938221 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:55.938203 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:30:56.003329 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.003527 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.003527 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.003527 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.003527 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003466 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.003527 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.003513 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgth\" (UniqueName: \"kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104592 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgth\" (UniqueName: \"kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.104749 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.104744 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.105085 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.105028 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.105179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.105119 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.105179 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.105148 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.107199 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.107176 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.107374 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.107357 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.112860 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.112841 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgth\" (UniqueName: \"kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth\") pod \"test-model-7aa53b-kserve-57d8f48bb7-47k4s\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.240286 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.240199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:30:56.374347 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.374323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:30:56.376585 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:30:56.376558 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ac936b8_31e5_41bf_9c56_5534fc408efa.slice/crio-e82e2f30da99168276c9c6f431528a2d90375924f0c3c788f8148e11c4c42283 WatchSource:0}: Error finding container e82e2f30da99168276c9c6f431528a2d90375924f0c3c788f8148e11c4c42283: Status 404 returned error can't find the container with id e82e2f30da99168276c9c6f431528a2d90375924f0c3c788f8148e11c4c42283 Jul 09 04:30:56.913439 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.913403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" event={"ID":"2ac936b8-31e5-41bf-9c56-5534fc408efa","Type":"ContainerStarted","Data":"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4"} Jul 09 04:30:56.913628 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.913456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" event={"ID":"2ac936b8-31e5-41bf-9c56-5534fc408efa","Type":"ContainerStarted","Data":"e82e2f30da99168276c9c6f431528a2d90375924f0c3c788f8148e11c4c42283"} Jul 09 04:30:56.930858 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:56.930808 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" podStartSLOduration=1.930792893 podStartE2EDuration="1.930792893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:56.930595208 +0000 UTC m=+1843.045772247" watchObservedRunningTime="2026-07-09 04:30:56.930792893 +0000 UTC m=+1843.045969928" Jul 09 04:30:59.440808 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:30:59.440776 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:31:00.011083 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:00.011035 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:31:00.011283 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:00.011245 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" podUID="8bf9e2f3-7e78-411c-9334-729e861ee104" containerName="cleanup" containerID="cri-o://87ac7f7c7195b0009903dc9e70270e59e36089845dd3399a3aae343b9007be8d" gracePeriod=30 Jul 09 04:31:06.240551 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:06.240514 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:06.240962 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:06.240575 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:06.253248 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:06.253227 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:06.972758 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:06.972728 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:11.335678 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:11.335642 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:31:19.831772 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.831734 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:19.835539 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.835520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:19.839663 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.839634 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a259cf\"/\"default-dockercfg-wx2zb\"" Jul 09 04:31:19.839798 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.839699 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a259cf\"/\"test-model-a259cf-kserve-self-signed-certs\"" Jul 09 04:31:19.839872 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.839810 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a259cf\"/\"openshift-service-ca.crt\"" Jul 09 04:31:19.839944 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.839589 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a259cf\"/\"kube-root-ca.crt\"" Jul 09 04:31:19.842367 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:19.842344 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:20.003308 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.003537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.003537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.003537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqt2r\" (UniqueName: \"kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.003537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.003537 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.003503 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104477 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104477 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104477 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqt2r\" (UniqueName: \"kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104731 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104731 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104731 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104922 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104900 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104986 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104930 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.104986 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.104949 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.106977 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.106960 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.107255 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.107237 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.112718 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.112699 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqt2r\" (UniqueName: \"kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r\") pod \"test-model-a259cf-kserve-675578b44f-knj6s\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.145642 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.145609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:20.281617 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:20.281590 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:20.283902 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:31:20.283875 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f8b6ce4_5dc1_4be7_8487_e870a3f0a972.slice/crio-a0b64264cfe798b5c143832f0731634e3013e5063366f2be93afaf7f92b51adc WatchSource:0}: Error finding container a0b64264cfe798b5c143832f0731634e3013e5063366f2be93afaf7f92b51adc: Status 404 returned error can't find the container with id a0b64264cfe798b5c143832f0731634e3013e5063366f2be93afaf7f92b51adc Jul 09 04:31:21.027378 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:21.027342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" event={"ID":"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972","Type":"ContainerStarted","Data":"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7"} Jul 09 04:31:21.027808 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:21.027388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" event={"ID":"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972","Type":"ContainerStarted","Data":"a0b64264cfe798b5c143832f0731634e3013e5063366f2be93afaf7f92b51adc"} Jul 09 04:31:21.045086 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:21.045022 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" podStartSLOduration=2.045005085 podStartE2EDuration="2.045005085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:31:21.043914431 +0000 UTC m=+1867.159091468" watchObservedRunningTime="2026-07-09 04:31:21.045005085 +0000 UTC m=+1867.160182099" Jul 09 04:31:26.858832 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:26.858795 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:31:30.072829 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.072779 2570 generic.go:356] "Generic (PLEG): container finished" podID="8bf9e2f3-7e78-411c-9334-729e861ee104" containerID="87ac7f7c7195b0009903dc9e70270e59e36089845dd3399a3aae343b9007be8d" exitCode=137 Jul 09 04:31:30.073188 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.072830 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" event={"ID":"8bf9e2f3-7e78-411c-9334-729e861ee104","Type":"ContainerDied","Data":"87ac7f7c7195b0009903dc9e70270e59e36089845dd3399a3aae343b9007be8d"} Jul 09 04:31:30.146339 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.146304 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:30.146521 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.146360 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:30.159800 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.159774 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:30.184465 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.184441 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:31:30.282928 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.282824 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv687\" (UniqueName: \"kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687\") pod \"8bf9e2f3-7e78-411c-9334-729e861ee104\" (UID: \"8bf9e2f3-7e78-411c-9334-729e861ee104\") " Jul 09 04:31:30.285101 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.285078 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687" (OuterVolumeSpecName: "kube-api-access-pv687") pod "8bf9e2f3-7e78-411c-9334-729e861ee104" (UID: "8bf9e2f3-7e78-411c-9334-729e861ee104"). InnerVolumeSpecName "kube-api-access-pv687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:31:30.383963 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:30.383923 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv687\" (UniqueName: \"kubernetes.io/projected/8bf9e2f3-7e78-411c-9334-729e861ee104-kube-api-access-pv687\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:31.079366 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.079326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" event={"ID":"8bf9e2f3-7e78-411c-9334-729e861ee104","Type":"ContainerDied","Data":"ce6e8bc9c0baf66aa2d4a8996a29faf26c1aa0fde240f42d908c295c4e62f683"} Jul 09 04:31:31.079366 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.079361 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv" Jul 09 04:31:31.079902 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.079428 2570 scope.go:117] "RemoveContainer" containerID="87ac7f7c7195b0009903dc9e70270e59e36089845dd3399a3aae343b9007be8d" Jul 09 04:31:31.092700 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.092674 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:31.095354 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.095332 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:31:31.099242 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:31.099222 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726190-5jdfv"] Jul 09 04:31:32.427127 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:32.427098 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf9e2f3-7e78-411c-9334-729e861ee104" path="/var/lib/kubelet/pods/8bf9e2f3-7e78-411c-9334-729e861ee104/volumes" Jul 09 04:31:41.685434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:41.685353 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:31:42.650127 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:42.650058 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:47.637570 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:47.637539 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:31:56.705702 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:56.705643 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" podUID="2ac936b8-31e5-41bf-9c56-5534fc408efa" containerName="main" containerID="cri-o://81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4" gracePeriod=45 Jul 09 04:31:56.964142 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:56.964086 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:57.120012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.119976 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120016 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fgth\" (UniqueName: \"kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120071 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120105 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120127 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120237 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120193 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs\") pod \"2ac936b8-31e5-41bf-9c56-5534fc408efa\" (UID: \"2ac936b8-31e5-41bf-9c56-5534fc408efa\") " Jul 09 04:31:57.120409 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120383 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:57.120409 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120395 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home" (OuterVolumeSpecName: "home") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:57.120519 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.120479 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:57.122437 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.122407 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm" (OuterVolumeSpecName: "dshm") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:57.122556 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.122461 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:31:57.122556 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.122498 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth" (OuterVolumeSpecName: "kube-api-access-4fgth") pod "2ac936b8-31e5-41bf-9c56-5534fc408efa" (UID: "2ac936b8-31e5-41bf-9c56-5534fc408efa"). InnerVolumeSpecName "kube-api-access-4fgth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:31:57.205719 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.205680 2570 generic.go:356] "Generic (PLEG): container finished" podID="2ac936b8-31e5-41bf-9c56-5534fc408efa" containerID="81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4" exitCode=0 Jul 09 04:31:57.205901 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.205728 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" Jul 09 04:31:57.205901 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.205775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" event={"ID":"2ac936b8-31e5-41bf-9c56-5534fc408efa","Type":"ContainerDied","Data":"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4"} Jul 09 04:31:57.205901 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.205824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s" event={"ID":"2ac936b8-31e5-41bf-9c56-5534fc408efa","Type":"ContainerDied","Data":"e82e2f30da99168276c9c6f431528a2d90375924f0c3c788f8148e11c4c42283"} Jul 09 04:31:57.205901 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.205846 2570 scope.go:117] "RemoveContainer" containerID="81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4" Jul 09 04:31:57.215480 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.215462 2570 scope.go:117] "RemoveContainer" containerID="81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4" Jul 09 04:31:57.215740 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:31:57.215722 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4\": container with ID starting with 81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4 not found: ID does not exist" containerID="81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4" Jul 09 04:31:57.215800 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.215753 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4"} err="failed to get container status \"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4\": rpc error: code = NotFound desc = could not find container \"81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4\": container with ID starting with 81c020c9417b454a2729edea77594de3d53c32149d5ec59ab8369c6312a900c4 not found: ID does not exist" Jul 09 04:31:57.221531 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221511 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac936b8-31e5-41bf-9c56-5534fc408efa-tls-certs\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.221531 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221532 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-dshm\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.221658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221542 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgth\" (UniqueName: \"kubernetes.io/projected/2ac936b8-31e5-41bf-9c56-5534fc408efa-kube-api-access-4fgth\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.221658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221551 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-model-cache\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.221658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221560 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-home\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.221658 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.221567 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac936b8-31e5-41bf-9c56-5534fc408efa-tmp-dir\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:57.227373 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.227350 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:31:57.230903 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.230884 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7aa53b/test-model-7aa53b-kserve-57d8f48bb7-47k4s"] Jul 09 04:31:57.671298 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.671254 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" podUID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" containerName="main" containerID="cri-o://9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7" gracePeriod=45 Jul 09 04:31:57.921406 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:57.921341 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:58.028663 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028605 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.028663 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028667 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.028931 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028700 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.028931 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028815 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqt2r\" (UniqueName: \"kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.028931 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028853 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.028931 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028919 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home\") pod \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\" (UID: \"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972\") " Jul 09 04:31:58.029167 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028954 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:58.029167 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.028966 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:58.029282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.029167 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tmp-dir\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.029282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.029187 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-model-cache\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.029282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.029201 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home" (OuterVolumeSpecName: "home") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:58.031109 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.031077 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm" (OuterVolumeSpecName: "dshm") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 04:31:58.031109 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.031098 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 04:31:58.031257 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.031139 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r" (OuterVolumeSpecName: "kube-api-access-bqt2r") pod "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" (UID: "4f8b6ce4-5dc1-4be7-8487-e870a3f0a972"). InnerVolumeSpecName "kube-api-access-bqt2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 04:31:58.129738 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.129696 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqt2r\" (UniqueName: \"kubernetes.io/projected/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-kube-api-access-bqt2r\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.129738 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.129731 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-dshm\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.129738 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.129743 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-home\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.129965 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.129751 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972-tls-certs\") on node \"ip-10-0-141-199.ec2.internal\" DevicePath \"\"" Jul 09 04:31:58.212031 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.211948 2570 generic.go:356] "Generic (PLEG): container finished" podID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" containerID="9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7" exitCode=0 Jul 09 04:31:58.212031 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.211993 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" Jul 09 04:31:58.212283 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.212033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" event={"ID":"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972","Type":"ContainerDied","Data":"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7"} Jul 09 04:31:58.212283 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.212093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s" event={"ID":"4f8b6ce4-5dc1-4be7-8487-e870a3f0a972","Type":"ContainerDied","Data":"a0b64264cfe798b5c143832f0731634e3013e5063366f2be93afaf7f92b51adc"} Jul 09 04:31:58.212283 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.212116 2570 scope.go:117] "RemoveContainer" containerID="9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7" Jul 09 04:31:58.221944 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.221872 2570 scope.go:117] "RemoveContainer" containerID="9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7" Jul 09 04:31:58.222223 ip-10-0-141-199 kubenswrapper[2570]: E0709 04:31:58.222204 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7\": container with ID starting with 9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7 not found: ID does not exist" containerID="9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7" Jul 09 04:31:58.222278 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.222232 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7"} err="failed to get container status \"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7\": rpc error: code = NotFound desc = could not find container \"9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7\": container with ID starting with 9f367ebb0320099f8e7c157167984bd65f17e5fa30c4262581de50d3808381f7 not found: ID does not exist" Jul 09 04:31:58.233278 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.233254 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:58.240462 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.240440 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a259cf/test-model-a259cf-kserve-675578b44f-knj6s"] Jul 09 04:31:58.427964 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.427927 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac936b8-31e5-41bf-9c56-5534fc408efa" path="/var/lib/kubelet/pods/2ac936b8-31e5-41bf-9c56-5534fc408efa/volumes" Jul 09 04:31:58.428410 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:31:58.428389 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" path="/var/lib/kubelet/pods/4f8b6ce4-5dc1-4be7-8487-e870a3f0a972/volumes" Jul 09 04:32:28.839169 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:32:28.839131 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:32:41.437023 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:32:41.436987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:32:54.130468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:32:54.130430 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:33:08.730165 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:33:08.730082 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:33:20.248548 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:33:20.248513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:33:32.949468 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:33:32.949436 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:35:14.480814 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:35:14.480787 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:35:14.482375 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:35:14.482357 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:36:36.532274 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:36.532237 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:36:46.800633 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:46.800604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bbc96db9-mpdf4_3763028c-44e2-4ab3-8b3e-7af799e783c5/authorino/0.log" Jul 09 04:36:48.630625 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:48.630587 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m5c6b"] Jul 09 04:36:51.793472 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:51.793443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbfc7556c-z26kh_f06f6af2-5610-4147-8b3e-63261b243bce/manager/0.log" Jul 09 04:36:52.284686 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:52.284657 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-pr977_9b1cc303-74fd-4bd4-abc5-2c93319ea408/copy-manifests/0.log" Jul 09 04:36:52.291017 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:52.290990 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-pr977_9b1cc303-74fd-4bd4-abc5-2c93319ea408/manager/0.log" Jul 09 04:36:53.263582 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.263545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/extract/0.log" Jul 09 04:36:53.268841 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.268821 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/util/0.log" Jul 09 04:36:53.274106 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.274083 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/pull/0.log" Jul 09 04:36:53.396073 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.396015 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/util/0.log" Jul 09 04:36:53.402052 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.402019 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/pull/0.log" Jul 09 04:36:53.407502 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.407487 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/extract/0.log" Jul 09 04:36:53.528434 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.528358 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/util/0.log" Jul 09 04:36:53.534545 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.534525 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/pull/0.log" Jul 09 04:36:53.540248 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.540232 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/extract/0.log" Jul 09 04:36:53.662726 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.662693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/util/0.log" Jul 09 04:36:53.668857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.668836 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/pull/0.log" Jul 09 04:36:53.674803 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.674787 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/extract/0.log" Jul 09 04:36:53.799758 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:53.799681 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bbc96db9-mpdf4_3763028c-44e2-4ab3-8b3e-7af799e783c5/authorino/0.log" Jul 09 04:36:54.158151 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:54.158127 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-88fvl_eb79263d-966a-4b1c-a3fd-6a158440a519/kuadrant-console-plugin/0.log" Jul 09 04:36:54.554939 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:54.554865 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m5c6b_8c0d9744-f573-4a22-bff5-ab017b94c7bb/limitador/0.log" Jul 09 04:36:55.408117 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:55.408081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-tq2gm_087b5914-a55e-4c86-8503-303121798881/kube-auth-proxy/0.log" Jul 09 04:36:56.872977 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:36:56.872944 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cxfxk_a064056d-884b-4cb6-b362-854a88a55ba9/main/0.log" Jul 09 04:37:01.117996 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.117962 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85fmr/must-gather-wnpft"] Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118296 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf9e2f3-7e78-411c-9334-729e861ee104" containerName="cleanup" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118313 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf9e2f3-7e78-411c-9334-729e861ee104" containerName="cleanup" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118337 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" containerName="main" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118348 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" containerName="main" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118375 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac936b8-31e5-41bf-9c56-5534fc408efa" containerName="main" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118383 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac936b8-31e5-41bf-9c56-5534fc408efa" containerName="main" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118469 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf9e2f3-7e78-411c-9334-729e861ee104" containerName="cleanup" Jul 09 04:37:01.118485 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118480 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac936b8-31e5-41bf-9c56-5534fc408efa" containerName="main" Jul 09 04:37:01.118865 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.118489 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8b6ce4-5dc1-4be7-8487-e870a3f0a972" containerName="main" Jul 09 04:37:01.121857 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.121839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.124821 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.124802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85fmr\"/\"kube-root-ca.crt\"" Jul 09 04:37:01.126006 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.125985 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-85fmr\"/\"openshift-service-ca.crt\"" Jul 09 04:37:01.126121 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.125985 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-85fmr\"/\"default-dockercfg-hslpv\"" Jul 09 04:37:01.130169 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.130146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/must-gather-wnpft"] Jul 09 04:37:01.200391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.200364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-must-gather-output\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.200391 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.200391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4t4p\" (UniqueName: \"kubernetes.io/projected/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-kube-api-access-z4t4p\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.300826 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.300788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-must-gather-output\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.300826 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.300823 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4t4p\" (UniqueName: \"kubernetes.io/projected/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-kube-api-access-z4t4p\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.301145 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.301124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-must-gather-output\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.309540 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.309519 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4t4p\" (UniqueName: \"kubernetes.io/projected/7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9-kube-api-access-z4t4p\") pod \"must-gather-wnpft\" (UID: \"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9\") " pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.432024 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.431994 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/must-gather-wnpft" Jul 09 04:37:01.563520 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.563496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/must-gather-wnpft"] Jul 09 04:37:01.565663 ip-10-0-141-199 kubenswrapper[2570]: W0709 04:37:01.565635 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec2279d_d1e9_4be6_afc4_485ac9c2cbb9.slice/crio-1f25ab80df8902568c0210bfe85a232aca3483d117f0eed61cc52c4ca061bc22 WatchSource:0}: Error finding container 1f25ab80df8902568c0210bfe85a232aca3483d117f0eed61cc52c4ca061bc22: Status 404 returned error can't find the container with id 1f25ab80df8902568c0210bfe85a232aca3483d117f0eed61cc52c4ca061bc22 Jul 09 04:37:01.567535 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.567515 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 04:37:01.600674 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:01.600639 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/must-gather-wnpft" event={"ID":"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9","Type":"ContainerStarted","Data":"1f25ab80df8902568c0210bfe85a232aca3483d117f0eed61cc52c4ca061bc22"} Jul 09 04:37:03.615612 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:03.615578 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/must-gather-wnpft" event={"ID":"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9","Type":"ContainerStarted","Data":"5432770f1a3bf156180bd40a28f9b786d0c2b6a1151b9d7b77fbc3ee92315383"} Jul 09 04:37:03.615997 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:03.615628 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/must-gather-wnpft" event={"ID":"7ec2279d-d1e9-4be6-afc4-485ac9c2cbb9","Type":"ContainerStarted","Data":"fdfd1d40cab42eca90c836b1eef2221e2a782d5bebc110b0805cc49fa0dc5cac"} Jul 09 04:37:03.632358 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:03.632292 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85fmr/must-gather-wnpft" podStartSLOduration=1.605496559 podStartE2EDuration="2.632270946s" totalImagesPullingTime="1.026774387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:37:03.629877052 +0000 UTC m=+2209.745054088" watchObservedRunningTime="2026-07-09 04:37:03.632270946 +0000 UTC m=+2209.747447983" Jul 09 04:37:04.131320 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:04.131289 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z872w_cb7a001d-b4c3-4aba-a69e-786b015d507a/global-pull-secret-syncer/0.log" Jul 09 04:37:04.242956 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:04.242923 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lm56f_b87c8df8-b0b7-417f-bf10-932250ae18cc/konnectivity-agent/0.log" Jul 09 04:37:04.299290 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:04.299259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-199.ec2.internal_91918a83cd21a5c27f7945240cf37d77/haproxy/0.log" Jul 09 04:37:08.229822 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.229750 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/extract/0.log" Jul 09 04:37:08.244787 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.244671 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/util/0.log" Jul 09 04:37:08.262217 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.262153 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dwmxh_72c15c87-5c92-485a-b902-ac3e0381639d/pull/0.log" Jul 09 04:37:08.288209 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.288187 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/extract/0.log" Jul 09 04:37:08.305012 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.304988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/util/0.log" Jul 09 04:37:08.322093 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.322035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mqlxt_fdbf64a9-a617-4e38-9343-ca71175a10a3/pull/0.log" Jul 09 04:37:08.343593 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.343531 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/extract/0.log" Jul 09 04:37:08.360452 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.360371 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/util/0.log" Jul 09 04:37:08.379679 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.379623 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zg57_89d942af-adc1-4508-9c8a-8da8eea0cfdb/pull/0.log" Jul 09 04:37:08.433210 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.433171 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/extract/0.log" Jul 09 04:37:08.482028 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.481950 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/util/0.log" Jul 09 04:37:08.520680 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.520655 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zbncx_00e88bca-f817-444e-8bc2-90244f80b2d9/pull/0.log" Jul 09 04:37:08.891167 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.891058 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7bbc96db9-mpdf4_3763028c-44e2-4ab3-8b3e-7af799e783c5/authorino/0.log" Jul 09 04:37:08.969106 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:08.969079 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-88fvl_eb79263d-966a-4b1c-a3fd-6a158440a519/kuadrant-console-plugin/0.log" Jul 09 04:37:09.211436 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:09.211407 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m5c6b_8c0d9744-f573-4a22-bff5-ab017b94c7bb/limitador/0.log" Jul 09 04:37:11.118145 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.118111 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mvt5n_b2c08bf2-554e-4749-8e40-084903a926e0/kube-state-metrics/0.log" Jul 09 04:37:11.135028 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.134988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mvt5n_b2c08bf2-554e-4749-8e40-084903a926e0/kube-rbac-proxy-main/0.log" Jul 09 04:37:11.155186 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.155151 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mvt5n_b2c08bf2-554e-4749-8e40-084903a926e0/kube-rbac-proxy-self/0.log" Jul 09 04:37:11.337778 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.337743 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h94cr_4fab6402-db93-4b93-be4f-41eaaff1265e/node-exporter/0.log" Jul 09 04:37:11.358619 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.358586 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h94cr_4fab6402-db93-4b93-be4f-41eaaff1265e/kube-rbac-proxy/0.log" Jul 09 04:37:11.377260 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.377185 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h94cr_4fab6402-db93-4b93-be4f-41eaaff1265e/init-textfile/0.log" Jul 09 04:37:11.717635 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.716542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rhl9w_1322387e-ffd8-421c-b6f1-1e387194680c/prometheus-operator/0.log" Jul 09 04:37:11.743642 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:11.743617 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rhl9w_1322387e-ffd8-421c-b6f1-1e387194680c/kube-rbac-proxy/0.log" Jul 09 04:37:14.433282 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.433249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-f74hh_bcd61866-271e-4b98-9092-63140cf94d74/download-server/0.log" Jul 09 04:37:14.694123 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.692961 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v"] Jul 09 04:37:14.699018 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.698992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.706001 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.705935 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v"] Jul 09 04:37:14.746606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.743823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-proc\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.746606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.743928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-sys\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.746606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.743973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-podres\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.746606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.744022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-lib-modules\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.746606 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.744085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cght\" (UniqueName: \"kubernetes.io/projected/45dd5552-912a-43de-9f42-572e864bb096-kube-api-access-7cght\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845595 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-proc\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845760 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-sys\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845760 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845662 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-proc\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845760 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845727 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-sys\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845760 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-podres\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845921 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-podres\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845921 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-lib-modules\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.845921 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cght\" (UniqueName: \"kubernetes.io/projected/45dd5552-912a-43de-9f42-572e864bb096-kube-api-access-7cght\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.846027 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.845918 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45dd5552-912a-43de-9f42-572e864bb096-lib-modules\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:14.854072 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:14.854028 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cght\" (UniqueName: \"kubernetes.io/projected/45dd5552-912a-43de-9f42-572e864bb096-kube-api-access-7cght\") pod \"perf-node-gather-daemonset-cbn7v\" (UID: \"45dd5552-912a-43de-9f42-572e864bb096\") " pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:15.014729 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:15.014641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:15.191167 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:15.191132 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v"] Jul 09 04:37:15.719210 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:15.719169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" event={"ID":"45dd5552-912a-43de-9f42-572e864bb096","Type":"ContainerStarted","Data":"e33d5ebe4e396450279087a30da11b7261b87da2c7806a0c54edd2d981c1a6aa"} Jul 09 04:37:15.719814 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:15.719788 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:15.719922 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:15.719828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" event={"ID":"45dd5552-912a-43de-9f42-572e864bb096","Type":"ContainerStarted","Data":"abcc4e4e18d1028039ab369a4e07056f526c94878dd15d768a68dee475c62030"} Jul 09 04:37:16.151807 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:16.151777 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnsj2_7075c303-5849-45fd-a889-7370137846d5/dns/0.log" Jul 09 04:37:16.170156 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:16.170124 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnsj2_7075c303-5849-45fd-a889-7370137846d5/kube-rbac-proxy/0.log" Jul 09 04:37:16.236213 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:16.236184 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zq56v_bcb69976-0b5d-41c7-97de-84e379767d58/dns-node-resolver/0.log" Jul 09 04:37:16.762646 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:16.762615 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64ddc67c5d-2vs8n_b9212353-2fa5-4c96-a873-995fef5359ce/registry/0.log" Jul 09 04:37:16.819271 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:16.819235 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzh4j_04568c63-abf8-4625-a487-478ab797e339/node-ca/0.log" Jul 09 04:37:17.892855 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:17.892822 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-tq2gm_087b5914-a55e-4c86-8503-303121798881/kube-auth-proxy/0.log" Jul 09 04:37:18.769862 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:18.769835 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b8brk_5b7fb9b2-8775-48f1-8a57-3068090257c4/serve-healthcheck-canary/0.log" Jul 09 04:37:19.491326 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:19.491302 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qkwhv_469c004e-31b2-446e-b6d3-e3067cd22c44/kube-rbac-proxy/0.log" Jul 09 04:37:19.508231 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:19.508200 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qkwhv_469c004e-31b2-446e-b6d3-e3067cd22c44/exporter/0.log" Jul 09 04:37:19.525590 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:19.525554 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qkwhv_469c004e-31b2-446e-b6d3-e3067cd22c44/extractor/0.log" Jul 09 04:37:21.738194 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:21.738163 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" Jul 09 04:37:21.759167 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:21.759116 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-85fmr/perf-node-gather-daemonset-cbn7v" podStartSLOduration=7.759099659 podStartE2EDuration="7.759099659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:37:15.736302445 +0000 UTC m=+2221.851479482" watchObservedRunningTime="2026-07-09 04:37:21.759099659 +0000 UTC m=+2227.874276688" Jul 09 04:37:21.887604 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:21.887567 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbfc7556c-z26kh_f06f6af2-5610-4147-8b3e-63261b243bce/manager/0.log" Jul 09 04:37:22.055244 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:22.055157 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-pr977_9b1cc303-74fd-4bd4-abc5-2c93319ea408/manager/0.log" Jul 09 04:37:22.071187 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:22.071160 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67dd5f9985-pr977_9b1cc303-74fd-4bd4-abc5-2c93319ea408/copy-manifests/0.log" Jul 09 04:37:23.366138 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:23.366063 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ff67c745d-hj2xb_0e6c8439-a172-4639-a898-f2ac7bff3557/manager/0.log" Jul 09 04:37:24.707004 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:24.706958 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-tq2gm_087b5914-a55e-4c86-8503-303121798881/kube-auth-proxy/0.log" Jul 09 04:37:26.229419 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:26.229383 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b68b5bb4-tq2gm_087b5914-a55e-4c86-8503-303121798881/kube-auth-proxy/0.log" Jul 09 04:37:29.972753 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:29.972727 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/kube-multus-additional-cni-plugins/0.log" Jul 09 04:37:29.989755 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:29.989727 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/egress-router-binary-copy/0.log" Jul 09 04:37:30.007247 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.007214 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/cni-plugins/0.log" Jul 09 04:37:30.032792 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.032764 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/bond-cni-plugin/0.log" Jul 09 04:37:30.052578 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.052541 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/routeoverride-cni/0.log" Jul 09 04:37:30.072276 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.072212 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/whereabouts-cni-bincopy/0.log" Jul 09 04:37:30.091496 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.091454 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82x8f_eeba12f8-f6d5-413f-b135-50720a67ab29/whereabouts-cni/0.log" Jul 09 04:37:30.271781 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.271699 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-drpsr_ab8cd9ab-b1e1-401e-ab33-c3c46a28d8ec/kube-multus/0.log" Jul 09 04:37:30.359405 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.359375 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r64cg_e3a6e221-92b6-4128-ae09-a15961f44acf/network-metrics-daemon/0.log" Jul 09 04:37:30.374243 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:30.374207 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r64cg_e3a6e221-92b6-4128-ae09-a15961f44acf/kube-rbac-proxy/0.log" Jul 09 04:37:31.547168 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.547080 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-controller/0.log" Jul 09 04:37:31.559965 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.559937 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/1.log" Jul 09 04:37:31.559965 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.559943 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovn-acl-logging/0.log" Jul 09 04:37:31.574913 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.574878 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/kube-rbac-proxy-node/0.log" Jul 09 04:37:31.601399 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.601373 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 04:37:31.618912 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.618882 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/northd/0.log" Jul 09 04:37:31.635443 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.635412 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/nbdb/0.log" Jul 09 04:37:31.652757 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.652724 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/sbdb/0.log" Jul 09 04:37:31.707064 ip-10-0-141-199 kubenswrapper[2570]: I0709 04:37:31.707013 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pr2n4_4b2ff288-75bd-49e2-a498-146c506d2b02/ovnkube-controller/0.log"