Jul 13 03:41:35.070246 ip-10-0-133-120 systemd[1]: Starting Kubernetes Kubelet... Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.499750 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506164 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506176 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506182 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506186 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506190 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506193 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506196 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:35.548793 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506199 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506202 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506205 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506208 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506211 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506213 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506217 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506220 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506222 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506225 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506228 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506230 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506233 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506236 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506239 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506241 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506244 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506247 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506249 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506252 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:35.562336 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506255 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506257 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506260 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506263 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506265 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506268 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506271 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506274 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506277 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506280 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506282 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506289 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506292 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506295 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506298 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506301 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506303 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506306 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506314 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506317 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:35.594058 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506319 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:35.573484 ip-10-0-133-120 systemd[1]: Started Kubernetes Kubelet. Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506322 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506325 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506327 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506330 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506333 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506336 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506338 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506341 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506343 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506346 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506349 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506351 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506354 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506356 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506358 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506361 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506364 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506366 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506369 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:35.626186 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506371 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506374 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506376 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506380 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506389 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506392 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506395 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506399 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506402 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506404 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506407 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506409 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506412 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506416 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506418 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506421 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506423 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506426 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506428 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:35.638362 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506431 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506433 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506436 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506440 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506443 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506446 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506449 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506452 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506454 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506457 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506460 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506463 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.506466 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507183 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507189 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507193 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507195 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507198 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507201 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:35.797024 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507204 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507207 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507210 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507213 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507216 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507219 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507221 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507224 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507226 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507229 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507232 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507234 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507237 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507239 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507242 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507244 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507247 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507249 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507252 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507254 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:35.797881 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507257 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507259 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507262 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507264 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507267 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507270 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507275 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507278 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507281 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507283 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507287 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507291 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507295 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507298 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507301 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507304 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507308 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507311 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:35.798851 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507314 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507317 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507320 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507322 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507325 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507327 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507330 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507332 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507335 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507337 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507340 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507342 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507345 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507347 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507350 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507353 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507355 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507358 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507361 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507363 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:35.799480 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507365 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507368 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507370 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507374 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507376 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507379 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507381 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507386 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507390 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507393 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507395 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507398 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507400 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507403 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507405 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507408 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507411 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507414 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507416 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507419 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:35.800132 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507421 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507424 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507427 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507429 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507432 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507434 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507439 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507441 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507444 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507446 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507449 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507451 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507454 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507456 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507459 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507808 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507812 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507815 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507817 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507822 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:35.801263 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507825 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507827 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507830 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507833 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507835 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507838 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507840 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507843 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507846 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507849 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507851 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507854 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507856 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507859 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507862 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507864 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507867 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507869 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507874 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507876 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:35.801870 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507879 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507881 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507884 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507886 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507889 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507891 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507894 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507898 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507901 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507903 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507906 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507910 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507913 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507916 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507919 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507921 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507924 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507926 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507929 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:35.999689 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507931 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507934 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507936 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507939 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507942 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507944 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507946 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507949 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507951 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507954 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507956 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507960 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507962 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507965 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507967 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507971 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507988 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507991 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507994 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507996 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:36.000481 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.507999 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508001 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508004 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508007 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508011 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508013 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508016 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508019 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508021 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508024 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508026 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508029 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508032 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508034 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508037 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508040 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508042 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508044 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508047 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508049 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:36.001124 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508052 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508054 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508058 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508062 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508065 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508070 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508073 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508076 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508078 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508081 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508084 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508086 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508089 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508091 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508094 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508162 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508173 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508181 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508188 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508196 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 03:41:36.001733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508200 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508203 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508213 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508216 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508219 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508223 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508226 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508229 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508232 2562 flags.go:64] FLAG: --cgroup-root="" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508235 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508238 2562 flags.go:64] FLAG: --client-ca-file="" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508240 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508243 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508248 2562 flags.go:64] FLAG: --cluster-domain="" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508250 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508255 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508258 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508261 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508265 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508268 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508271 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508274 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508277 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508280 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 03:41:36.002319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508283 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508286 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508289 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508293 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508297 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508300 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508303 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508306 2562 flags.go:64] FLAG: --enable-server="true" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508309 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508313 2562 flags.go:64] FLAG: --event-burst="100" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508316 2562 flags.go:64] FLAG: --event-qps="50" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508319 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508322 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508325 2562 flags.go:64] FLAG: --eviction-hard="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508328 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508331 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508334 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508337 2562 flags.go:64] FLAG: --eviction-soft="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508340 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508343 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508346 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508348 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508351 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508355 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508358 2562 flags.go:64] FLAG: --feature-gates="" Jul 13 03:41:36.003017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508362 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508365 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508368 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508371 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508374 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508377 2562 flags.go:64] FLAG: --help="false" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508380 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508383 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508386 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508389 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508392 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508397 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508400 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508403 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508406 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508409 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508412 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508416 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508419 2562 flags.go:64] FLAG: --kube-reserved="" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508422 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508424 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508427 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508430 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508433 2562 flags.go:64] FLAG: --lock-file="" Jul 13 03:41:36.003686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508435 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508438 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508441 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508446 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508449 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508452 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508456 2562 flags.go:64] FLAG: --logging-format="text" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508459 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508462 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508465 2562 flags.go:64] FLAG: --manifest-url="" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508468 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508472 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508475 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508478 2562 flags.go:64] FLAG: --max-pods="110" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508481 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508484 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508487 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508490 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508493 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508497 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508500 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508507 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508510 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508513 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 03:41:36.004362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508516 2562 flags.go:64] FLAG: --pod-cidr="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508519 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508525 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508528 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508531 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508534 2562 flags.go:64] FLAG: --port="10250" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508537 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508539 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0135fe73331f554b2" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508543 2562 flags.go:64] FLAG: --qos-reserved="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508545 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508548 2562 flags.go:64] FLAG: --register-node="true" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508551 2562 flags.go:64] FLAG: --register-with-taints="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508555 2562 flags.go:64] FLAG: --registry-burst="10" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508557 2562 flags.go:64] FLAG: --registry-qps="5" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508561 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508564 2562 flags.go:64] FLAG: --reserved-memory="" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508568 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508571 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508574 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508577 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508580 2562 flags.go:64] FLAG: --runonce="false" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508582 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508586 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508588 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508591 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 03:41:36.004970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508594 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508597 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508600 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508604 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508607 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508609 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508612 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508615 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508618 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508621 2562 flags.go:64] FLAG: --system-cgroups="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508624 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508629 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508632 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508635 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508639 2562 flags.go:64] FLAG: --tls-min-version="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508642 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508644 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508647 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508650 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508653 2562 flags.go:64] FLAG: --v="2" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508657 2562 flags.go:64] FLAG: --version="false" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508662 2562 flags.go:64] FLAG: --vmodule="" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508666 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.508669 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508780 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:36.005792 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508784 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508787 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508790 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508793 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508795 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508798 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508800 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508803 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508805 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508808 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508812 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508815 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508817 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508820 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508822 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508825 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508828 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508831 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508833 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:36.006622 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508836 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508839 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508841 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508844 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508846 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508850 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508853 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508856 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508859 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508863 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508866 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508868 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508871 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508873 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508876 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508878 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508881 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508884 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508886 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:36.007249 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508889 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508892 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508894 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508897 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508900 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508903 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508905 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508907 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508910 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508912 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508915 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508918 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508921 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508923 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508926 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508928 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508931 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508933 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508936 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508938 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:36.007861 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508941 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508943 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508947 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508950 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508952 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508955 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508957 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508960 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508962 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508965 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508967 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508970 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508987 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508990 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508992 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508995 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.508999 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509002 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509005 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509007 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:36.008475 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509011 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509015 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509018 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509021 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509024 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509027 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509030 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509034 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509037 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509040 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509043 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509046 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509049 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509052 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509056 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509058 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509061 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509064 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509066 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:36.009056 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.509069 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.509074 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.516558 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.516576 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.517272 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.517280 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517344 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517352 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517356 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517359 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517362 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517365 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517367 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517370 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:36.009604 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517373 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517376 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517379 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517381 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517384 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517386 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517389 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517394 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517396 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517399 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517401 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517404 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517407 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517410 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517412 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517416 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517419 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517422 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517424 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517427 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:36.010013 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517429 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517438 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517441 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517443 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517446 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517448 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517451 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517454 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517456 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517459 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517461 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517464 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517466 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517469 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517471 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517474 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517476 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517479 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517482 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:36.010562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517485 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517488 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517491 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517493 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517496 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517498 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517501 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517503 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517507 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517510 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517513 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517515 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517517 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517520 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517523 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517531 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517534 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517537 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517539 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517542 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:36.011108 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517545 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517548 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517550 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517553 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517555 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517559 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517563 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517566 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517569 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517571 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517574 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517576 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517580 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517582 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517585 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517588 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517590 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517593 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517595 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:36.011661 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517598 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517601 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517604 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517606 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517610 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517613 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517616 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517619 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517622 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517632 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517635 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517638 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517641 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.517646 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517808 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 03:41:36.012468 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517813 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517815 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517818 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517821 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517824 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517826 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517829 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517832 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517835 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517837 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517840 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517843 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517846 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517849 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517851 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517853 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517856 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517858 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517863 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517866 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 03:41:36.012911 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517869 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517871 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517874 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517876 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517879 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517881 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517884 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.517929 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518592 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518600 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518605 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518609 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518615 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518620 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518626 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518631 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518635 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518640 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518645 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518656 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 03:41:36.209694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518661 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518664 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518669 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518676 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518680 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518684 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518689 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518693 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518698 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518702 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518708 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518712 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518722 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518726 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518733 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518738 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518744 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518749 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518756 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 03:41:36.210917 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518762 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518790 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518795 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518800 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518809 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518815 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518819 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518825 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518830 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518834 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518838 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518843 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518847 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518851 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518855 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518878 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518884 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518893 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518897 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518902 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 03:41:36.211443 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518906 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518911 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518915 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518921 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518925 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518929 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518934 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518938 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518942 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518946 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518955 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518960 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518964 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.518969 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.519006 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.519011 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.519015 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.519020 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 03:41:36.212006 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:35.519024 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.519033 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.519532 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.523631 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.525004 2562 server.go:1019] "Starting client certificate rotation" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.525107 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.525153 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.527507 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.528644 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.539774 2562 log.go:25] "Validated CRI v1 runtime API" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.541964 2562 log.go:25] "Validated CRI v1 image API" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.542838 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 03:41:36.212488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.546758 2562 fs.go:135] Filesystem UUIDs: map[4666473d-9f0e-4e55-bb9e-997fb54460cf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f4bd2138-e03e-46bf-bd50-b38254991fc9:/dev/nvme0n1p3] Jul 13 03:41:36.212874 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.546773 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.552251 2562 manager.go:216] Machine: {Timestamp:2026-07-13 03:41:35.550874575 +0000 UTC m=+0.326832268 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072743 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb8c1677018fca6ea7b6b66613868 SystemUUID:ec2bb8c1-6770-18fc-a6ea-7b6b66613868 BootID:175f15b2-c259-4a69-a17a-4982a994c6b1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:55:d9:c1:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:55:d9:c1:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:97:14:38:d0:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.552344 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.552417 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.552853 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.555422 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.555456 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 03:41:36.212947 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.556172 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.556182 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.556190 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.556761 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.557604 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560275 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560383 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560408 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560417 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560430 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560439 2562 kubelet.go:398] "Adding apiserver pod source" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560450 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560756 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.560771 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.564999 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568078 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568110 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568747 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568762 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568774 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568784 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568798 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568807 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568816 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568823 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568831 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568840 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568853 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.568858 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.569549 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.569558 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.569736 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.569769 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572326 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572624 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdrkn" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572642 2562 server.go:1268] "Started kubelet" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572727 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572746 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.572795 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.574529 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 03:41:36.213263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.576077 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.579740 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdrkn" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.580053 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.580376 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.580386 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.580479 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581299 2562 factory.go:55] Registering systemd factory Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581307 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581323 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581333 2562 factory.go:223] Registration of the systemd container factory successfully Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581302 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581449 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581459 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581551 2562 factory.go:153] Registering CRI-O factory Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581562 2562 factory.go:223] Registration of the crio container factory successfully Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581601 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581617 2562 factory.go:103] Registering Raw factory Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.581628 2562 manager.go:1196] Started watching for new ooms in manager Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.582159 2562 manager.go:349] Starting recovery of all containers Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.584175 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.581136 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-120.ec2.internal.18c1bc6b186b209d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-120.ec2.internal,UID:ip-10-0-133-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-120.ec2.internal,},FirstTimestamp:2026-07-13 03:41:35.572623517 +0000 UTC m=+0.348581208,LastTimestamp:2026-07-13 03:41:35.572623517 +0000 UTC m=+0.348581208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-120.ec2.internal,}" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.594531 2562 manager.go:354] Recovery completed Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.596507 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-120.ec2.internal\" not found" node="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.599130 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601043 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601068 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601079 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601529 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601538 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601544 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.601553 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602315 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602331 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602341 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.214276 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602605 2562 policy_none.go:49] "None policy: Start" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602616 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.602626 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.603630 2562 policy_none.go:47] "Start" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664514 2562 manager.go:337] "Starting Device Plugin manager" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.664562 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664571 2562 server.go:87] "Starting device plugin registration server" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664767 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664780 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664871 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664950 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.664959 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.665563 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.665617 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.744840 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.746001 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.746046 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.746070 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.746108 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.748519 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.765769 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.766555 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.766587 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.766601 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.766622 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.774651 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.774669 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-120.ec2.internal\": node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.791868 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.846541 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"] Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.846731 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.847468 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.847497 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.847510 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.848770 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.848863 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.848886 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849447 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849457 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.215199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849471 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849484 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849488 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.849499 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.850605 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.850628 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.851323 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.851350 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.851366 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.869518 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-120.ec2.internal\" not found" node="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.876883 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-120.ec2.internal\" not found" node="ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.882207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.882227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.882244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa2a3eea528d01b7f1c87c2d7e7e408-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"7aa2a3eea528d01b7f1c87c2d7e7e408\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.892288 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.982927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa2a3eea528d01b7f1c87c2d7e7e408-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"7aa2a3eea528d01b7f1c87c2d7e7e408\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.983236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa2a3eea528d01b7f1c87c2d7e7e408-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"7aa2a3eea528d01b7f1c87c2d7e7e408\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.983269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.983284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.983309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:35.983331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84ea5db1b3c2451296d25e11b01000b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"84ea5db1b3c2451296d25e11b01000b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:35.992404 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.093177 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.171355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.180045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.216772 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.194215 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.406414 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.294788 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.406414 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.395303 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.496024 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.521848 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.523935 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.524056 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.561595 2562 apiserver.go:52] "Watching apiserver" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.579792 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.580150 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wpr6r","openshift-image-registry/node-ca-mg78z","openshift-multus/multus-additional-cni-plugins-jdnr6","openshift-multus/multus-cq9vg","openshift-multus/network-metrics-daemon-8ggwx","openshift-ovn-kubernetes/ovnkube-node-g5xf7","kube-system/konnectivity-agent-tjvh9","openshift-cluster-node-tuning-operator/tuned-xpjhw","openshift-network-diagnostics/network-check-target-pcmzm","openshift-network-operator/iptables-alerter-lfqj5","kube-system/global-pull-secret-syncer-45z9p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm"] Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.581155 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.581175 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.581583 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.589690 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.589714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.589896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.590023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.590134 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.590907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e93eb64b-9ff7-4ada-ad80-286e166db78a-hosts-file\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591091 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 03:36:36 +0000 UTC" deadline="2028-04-17 21:00:28.234499216 +0000 UTC" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591155 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15473h18m51.643348791s" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8kv\" (UniqueName: \"kubernetes.io/projected/e93eb64b-9ff7-4ada-ad80-286e166db78a-kube-api-access-fg8kv\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wh8\" (UniqueName: \"kubernetes.io/projected/55459b3f-e007-4218-9cf8-85f8f7bb9976-kube-api-access-r4wh8\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.602321 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93eb64b-9ff7-4ada-ad80-286e166db78a-tmp-dir\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55459b3f-e007-4218-9cf8-85f8f7bb9976-host\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.591425 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55459b3f-e007-4218-9cf8-85f8f7bb9976-serviceca\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.594214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.594255 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.594327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.594915 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.595800 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596080 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-flmxd\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596227 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596231 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596371 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596453 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596513 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6t8dz\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596524 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.596722 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597029 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vnxtb\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597066 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2c55k\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597156 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gglj\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.597914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.597967 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.598033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.598059 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4hpn8\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.598417 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.598443 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 03:41:36.604029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599655 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599796 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599851 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599898 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.599921 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.600117 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.600340 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.600438 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.600451 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lcgp2\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.601250 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601412 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601511 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xrngf\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601996 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.601964 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.602222 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.604117 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"] Jul 13 03:41:36.605638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.604195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.610689 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.610750 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.610786 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.610754 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vlw2c\"" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.612735 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.613059 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"] Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.619243 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn6m6" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.626603 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn6m6" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.682175 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.691918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-etc-kubernetes\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.691971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgmk\" (UniqueName: \"kubernetes.io/projected/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-kube-api-access-dbgmk\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-kubelet-config\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e93eb64b-9ff7-4ada-ad80-286e166db78a-hosts-file\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692058 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-systemd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-netd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-tmp\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692124 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78qc\" (UniqueName: \"kubernetes.io/projected/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-kube-api-access-m78qc\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e93eb64b-9ff7-4ada-ad80-286e166db78a-hosts-file\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.778815 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-cnibin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-k8s-cni-cncf-io\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-systemd-units\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-netns\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpstw\" (UniqueName: \"kubernetes.io/projected/7cf560ba-63be-4f6d-a5f9-5af6100690d0-kube-api-access-wpstw\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-host\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-device-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-multus\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-sys\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-tuned\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-socket-dir-parent\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-var-lib-kubelet\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55459b3f-e007-4218-9cf8-85f8f7bb9976-host\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-kubelet\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-etc-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692658 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55459b3f-e007-4218-9cf8-85f8f7bb9976-host\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-env-overrides\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-script-lib\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-modprobe-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-system-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-os-release\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-hostroot\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.780583 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-conf-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-log-socket\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-config\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93eb64b-9ff7-4ada-ad80-286e166db78a-tmp-dir\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692973 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-kubelet\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.692995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysconfig\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-run\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlr5d\" (UniqueName: \"kubernetes.io/projected/446ecfd0-eaac-47cd-b3a8-6046826dcbac-kube-api-access-qlr5d\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db23e086-5d21-4d81-9396-72be63d48a33-konnectivity-ca\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-iptables-alerter-script\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wh8\" (UniqueName: \"kubernetes.io/projected/55459b3f-e007-4218-9cf8-85f8f7bb9976-kube-api-access-r4wh8\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.781153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93eb64b-9ff7-4ada-ad80-286e166db78a-tmp-dir\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lnx\" (UniqueName: \"kubernetes.io/projected/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-kube-api-access-k4lnx\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-cni-binary-copy\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-daemon-config\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693425 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-kubernetes\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-systemd\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-registration-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-ovn\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-node-log\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-bin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovn-node-metrics-cert\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-lib-modules\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-sys-fs\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cnibin\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.781671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-netns\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9dp\" (UniqueName: \"kubernetes.io/projected/e22f94d2-04ca-470e-8664-5c33f1ab1691-kube-api-access-rd9dp\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-conf\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693771 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8kv\" (UniqueName: \"kubernetes.io/projected/e93eb64b-9ff7-4ada-ad80-286e166db78a-kube-api-access-fg8kv\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-os-release\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-var-lib-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-bin\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55459b3f-e007-4218-9cf8-85f8f7bb9976-serviceca\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.693947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-system-cni-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-slash\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:36.782314 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db23e086-5d21-4d81-9396-72be63d48a33-agent-certs\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-host-slash\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-dbus\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-multus-certs\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-socket-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmnw\" (UniqueName: \"kubernetes.io/projected/103c32dd-423d-49a5-806f-11e8075c71ac-kube-api-access-8kmnw\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.694475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55459b3f-e007-4218-9cf8-85f8f7bb9976-serviceca\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.697923 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.701719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8kv\" (UniqueName: \"kubernetes.io/projected/e93eb64b-9ff7-4ada-ad80-286e166db78a-kube-api-access-fg8kv\") pod \"node-resolver-wpr6r\" (UID: \"e93eb64b-9ff7-4ada-ad80-286e166db78a\") " pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.701729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wh8\" (UniqueName: \"kubernetes.io/projected/55459b3f-e007-4218-9cf8-85f8f7bb9976-kube-api-access-r4wh8\") pod \"node-ca-mg78z\" (UID: \"55459b3f-e007-4218-9cf8-85f8f7bb9976\") " pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:36.718277 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ea5db1b3c2451296d25e11b01000b7.slice/crio-e9938f2078d04f46f40c9d2639bbcc8e82a263711c31b4607b397f1062d714ca WatchSource:0}: Error finding container e9938f2078d04f46f40c9d2639bbcc8e82a263711c31b4607b397f1062d714ca: Status 404 returned error can't find the container with id e9938f2078d04f46f40c9d2639bbcc8e82a263711c31b4607b397f1062d714ca Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:36.718490 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2a3eea528d01b7f1c87c2d7e7e408.slice/crio-072ce63cabfe67d91a6149e472d22a2e83ae29cd06933a2f5387e6fc3746be73 WatchSource:0}: Error finding container 072ce63cabfe67d91a6149e472d22a2e83ae29cd06933a2f5387e6fc3746be73: Status 404 returned error can't find the container with id 072ce63cabfe67d91a6149e472d22a2e83ae29cd06933a2f5387e6fc3746be73 Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.723802 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 03:41:36.782817 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.749515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" event={"ID":"7aa2a3eea528d01b7f1c87c2d7e7e408","Type":"ContainerStarted","Data":"072ce63cabfe67d91a6149e472d22a2e83ae29cd06933a2f5387e6fc3746be73"} Jul 13 03:41:36.783421 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.750500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"84ea5db1b3c2451296d25e11b01000b7","Type":"ContainerStarted","Data":"e9938f2078d04f46f40c9d2639bbcc8e82a263711c31b4607b397f1062d714ca"} Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db23e086-5d21-4d81-9396-72be63d48a33-konnectivity-ca\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-iptables-alerter-script\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lnx\" (UniqueName: \"kubernetes.io/projected/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-kube-api-access-k4lnx\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-cni-binary-copy\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-daemon-config\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-kubernetes\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-systemd\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-registration-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-kubernetes\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-registration-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-ovn\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.979176 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-systemd\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-node-log\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-ovn\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-bin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-node-log\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovn-node-metrics-cert\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db23e086-5d21-4d81-9396-72be63d48a33-konnectivity-ca\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-lib-modules\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-bin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-sys-fs\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cnibin\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-netns\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795761 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-sys-fs\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9dp\" (UniqueName: \"kubernetes.io/projected/e22f94d2-04ca-470e-8664-5c33f1ab1691-kube-api-access-rd9dp\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795789 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cnibin\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.980747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795796 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-lib-modules\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-conf\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-netns\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-os-release\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-iptables-alerter-script\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.795909 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-daemon-config\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-var-lib-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.795954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e22f94d2-04ca-470e-8664-5c33f1ab1691-cni-binary-copy\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.795982 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:41:37.295946939 +0000 UTC m=+2.071904625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-conf\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-var-lib-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-bin\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-os-release\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-bin\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-system-cni-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-slash\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-system-cni-dir\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-slash\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db23e086-5d21-4d81-9396-72be63d48a33-agent-certs\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-host-slash\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.796346 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-dbus\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.981848 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-host-slash\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.796397 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:41:37.296377422 +0000 UTC m=+2.072335103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-multus-certs\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-socket-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmnw\" (UniqueName: \"kubernetes.io/projected/103c32dd-423d-49a5-806f-11e8075c71ac-kube-api-access-8kmnw\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-multus-certs\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-etc-kubernetes\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-dbus\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgmk\" (UniqueName: \"kubernetes.io/projected/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-kube-api-access-dbgmk\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-kubelet-config\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-etc-kubernetes\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-systemd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-socket-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-netd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-systemd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-tmp\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbe98aea-97ba-462c-b92d-4d147e56fd4d-kubelet-config\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-cni-netd\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78qc\" (UniqueName: \"kubernetes.io/projected/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-kube-api-access-m78qc\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-cnibin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-k8s-cni-cncf-io\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-systemd-units\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.796993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-netns\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpstw\" (UniqueName: \"kubernetes.io/projected/7cf560ba-63be-4f6d-a5f9-5af6100690d0-kube-api-access-wpstw\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.982978 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-host\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-cnibin\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-run-k8s-cni-cncf-io\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797103 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-systemd-units\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797152 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-host\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-netns\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-device-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/103c32dd-423d-49a5-806f-11e8075c71ac-device-dir\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-multus\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-sys\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-tuned\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-cni-multus\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-sys\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-socket-dir-parent\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:36.983811 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-var-lib-kubelet\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-socket-dir-parent\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-kubelet\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-var-lib-kubelet\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-etc-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-host-var-lib-kubelet\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-run-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-etc-openvswitch\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-env-overrides\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.189862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-script-lib\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-modprobe-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-system-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-os-release\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-hostroot\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-conf-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.797989 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-modprobe-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-log-socket\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-config\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-kubelet\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysconfig\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-run\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr5d\" (UniqueName: \"kubernetes.io/projected/446ecfd0-eaac-47cd-b3a8-6046826dcbac-kube-api-access-qlr5d\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-env-overrides\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-system-cni-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysconfig\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-os-release\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-host-kubelet\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-hostroot\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e22f94d2-04ca-470e-8664-5c33f1ab1691-multus-conf-dir\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cf560ba-63be-4f6d-a5f9-5af6100690d0-log-socket\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-sysctl-d\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798619 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-run\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovn-node-metrics-cert\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.192862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798841 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-config\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.798856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cf560ba-63be-4f6d-a5f9-5af6100690d0-ovnkube-script-lib\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.799145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db23e086-5d21-4d81-9396-72be63d48a33-agent-certs\") pod \"konnectivity-agent-tjvh9\" (UID: \"db23e086-5d21-4d81-9396-72be63d48a33\") " pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.799308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-tmp\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.799586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-etc-tuned\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.805166 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.805196 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.805225 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:36.805296 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:41:37.305276098 +0000 UTC m=+2.081233807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.810524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9dp\" (UniqueName: \"kubernetes.io/projected/e22f94d2-04ca-470e-8664-5c33f1ab1691-kube-api-access-rd9dp\") pod \"multus-cq9vg\" (UID: \"e22f94d2-04ca-470e-8664-5c33f1ab1691\") " pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.811177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgmk\" (UniqueName: \"kubernetes.io/projected/611bfda6-ecf4-4294-bf2a-3b38e60dfde5-kube-api-access-dbgmk\") pod \"tuned-xpjhw\" (UID: \"611bfda6-ecf4-4294-bf2a-3b38e60dfde5\") " pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.811194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78qc\" (UniqueName: \"kubernetes.io/projected/c6e3f1a3-8dc2-4924-9eaf-c279e6940930-kube-api-access-m78qc\") pod \"iptables-alerter-lfqj5\" (UID: \"c6e3f1a3-8dc2-4924-9eaf-c279e6940930\") " pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.811269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmnw\" (UniqueName: \"kubernetes.io/projected/103c32dd-423d-49a5-806f-11e8075c71ac-kube-api-access-8kmnw\") pod \"aws-ebs-csi-driver-node-p9jtm\" (UID: \"103c32dd-423d-49a5-806f-11e8075c71ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.811372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr5d\" (UniqueName: \"kubernetes.io/projected/446ecfd0-eaac-47cd-b3a8-6046826dcbac-kube-api-access-qlr5d\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:37.193395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.813084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lnx\" (UniqueName: \"kubernetes.io/projected/4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd-kube-api-access-k4lnx\") pod \"multus-additional-cni-plugins-jdnr6\" (UID: \"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd\") " pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.815771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpstw\" (UniqueName: \"kubernetes.io/projected/7cf560ba-63be-4f6d-a5f9-5af6100690d0-kube-api-access-wpstw\") pod \"ovnkube-node-g5xf7\" (UID: \"7cf560ba-63be-4f6d-a5f9-5af6100690d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.921071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wpr6r" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:36.929699 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93eb64b_9ff7_4ada_ad80_286e166db78a.slice/crio-8e8f05c6585f74576dd3b0d4f08a5abf2269e982f812ef7586753dac174a3d38 WatchSource:0}: Error finding container 8e8f05c6585f74576dd3b0d4f08a5abf2269e982f812ef7586753dac174a3d38: Status 404 returned error can't find the container with id 8e8f05c6585f74576dd3b0d4f08a5abf2269e982f812ef7586753dac174a3d38 Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.948527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mg78z" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.961027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:36.968452 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fba8a3f_e72d_4de0_9c38_a4c9830ff7fd.slice/crio-d6845ef804c1afd847070c1fa951bf92e3932debdec6786f657ba479ddf9f728 WatchSource:0}: Error finding container d6845ef804c1afd847070c1fa951bf92e3932debdec6786f657ba479ddf9f728: Status 404 returned error can't find the container with id d6845ef804c1afd847070c1fa951bf92e3932debdec6786f657ba479ddf9f728 Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.970384 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.973797 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq9vg" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:36.982263 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22f94d2_04ca_470e_8664_5c33f1ab1691.slice/crio-a8db6705d9a824d356625a52d3793c8216e67885d857537a51f53eec580f2ca6 WatchSource:0}: Error finding container a8db6705d9a824d356625a52d3793c8216e67885d857537a51f53eec580f2ca6: Status 404 returned error can't find the container with id a8db6705d9a824d356625a52d3793c8216e67885d857537a51f53eec580f2ca6 Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:36.996666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:37.003603 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb23e086_5d21_4d81_9396_72be63d48a33.slice/crio-ef76d199ed8e9d77a1f6c04150c5d544e402a432716bf8ff7b69598a1628ef5b WatchSource:0}: Error finding container ef76d199ed8e9d77a1f6c04150c5d544e402a432716bf8ff7b69598a1628ef5b: Status 404 returned error can't find the container with id ef76d199ed8e9d77a1f6c04150c5d544e402a432716bf8ff7b69598a1628ef5b Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.030086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.036073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:37.036778 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf560ba_63be_4f6d_a5f9_5af6100690d0.slice/crio-49650c5b040033611c98543eb99c151359f85f9c2b32975e54ae1e1d4f3e1b31 WatchSource:0}: Error finding container 49650c5b040033611c98543eb99c151359f85f9c2b32975e54ae1e1d4f3e1b31: Status 404 returned error can't find the container with id 49650c5b040033611c98543eb99c151359f85f9c2b32975e54ae1e1d4f3e1b31 Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.046248 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfqj5" Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:37.046247 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611bfda6_ecf4_4294_bf2a_3b38e60dfde5.slice/crio-cd4d6a45a6bcecc2eedae101d41daa35f3507a24c05c484dd6458292f781391e WatchSource:0}: Error finding container cd4d6a45a6bcecc2eedae101d41daa35f3507a24c05c484dd6458292f781391e: Status 404 returned error can't find the container with id cd4d6a45a6bcecc2eedae101d41daa35f3507a24c05c484dd6458292f781391e Jul 13 03:41:37.193864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.049887 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" Jul 13 03:41:37.194450 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:41:37.057415 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e3f1a3_8dc2_4924_9eaf_c279e6940930.slice/crio-5016514ef128c4afa22c14a99f865b30f76eef1f4669cbc8b437014e319498bb WatchSource:0}: Error finding container 5016514ef128c4afa22c14a99f865b30f76eef1f4669cbc8b437014e319498bb: Status 404 returned error can't find the container with id 5016514ef128c4afa22c14a99f865b30f76eef1f4669cbc8b437014e319498bb Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.303119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.303167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.303300 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.303368 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:41:38.303347912 +0000 UTC m=+3.079305604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.303407 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:37.344014 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.303474 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:41:38.303454335 +0000 UTC m=+3.079412018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:37.410630 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.404347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:37.410630 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.404528 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:37.410630 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.404551 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:37.410630 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.404562 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:37.410630 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.404613 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:41:38.404598486 +0000 UTC m=+3.180556166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:37.640470 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.628260 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 03:36:37 +0000 UTC" deadline="2028-01-13 10:18:39.218181249 +0000 UTC" Jul 13 03:41:37.640470 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.628318 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13182h37m1.589867988s" Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.749643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:37.749771 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.757437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" event={"ID":"103c32dd-423d-49a5-806f-11e8075c71ac","Type":"ContainerStarted","Data":"92d6fb23e1464b73005d7c440666257635913dacd8517b964780697aec5f3649"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.783437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfqj5" event={"ID":"c6e3f1a3-8dc2-4924-9eaf-c279e6940930","Type":"ContainerStarted","Data":"5016514ef128c4afa22c14a99f865b30f76eef1f4669cbc8b437014e319498bb"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.791383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjvh9" event={"ID":"db23e086-5d21-4d81-9396-72be63d48a33","Type":"ContainerStarted","Data":"ef76d199ed8e9d77a1f6c04150c5d544e402a432716bf8ff7b69598a1628ef5b"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.794688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerStarted","Data":"d6845ef804c1afd847070c1fa951bf92e3932debdec6786f657ba479ddf9f728"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.806439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mg78z" event={"ID":"55459b3f-e007-4218-9cf8-85f8f7bb9976","Type":"ContainerStarted","Data":"11e4a964f98c451925da539ad3a55e867e8bbda6ba9b9a4ca84f245cfd888067"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.809601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wpr6r" event={"ID":"e93eb64b-9ff7-4ada-ad80-286e166db78a","Type":"ContainerStarted","Data":"8e8f05c6585f74576dd3b0d4f08a5abf2269e982f812ef7586753dac174a3d38"} Jul 13 03:41:37.823979 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.813377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" event={"ID":"611bfda6-ecf4-4294-bf2a-3b38e60dfde5","Type":"ContainerStarted","Data":"cd4d6a45a6bcecc2eedae101d41daa35f3507a24c05c484dd6458292f781391e"} Jul 13 03:41:37.831218 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.829128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"49650c5b040033611c98543eb99c151359f85f9c2b32975e54ae1e1d4f3e1b31"} Jul 13 03:41:37.833969 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:37.831598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq9vg" event={"ID":"e22f94d2-04ca-470e-8664-5c33f1ab1691","Type":"ContainerStarted","Data":"a8db6705d9a824d356625a52d3793c8216e67885d857537a51f53eec580f2ca6"} Jul 13 03:41:38.089621 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.089592 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.312300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.312362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.312538 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.312602 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:41:40.312583227 +0000 UTC m=+5.088540907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.313036 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:38.315124 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.313103 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:41:40.31308608 +0000 UTC m=+5.089043806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:38.413563 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.413519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:38.413761 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.413734 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:38.413761 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.413755 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:38.413889 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.413769 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:38.413889 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.413840 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:41:40.413818785 +0000 UTC m=+5.189776489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:38.628913 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.628819 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 03:36:37 +0000 UTC" deadline="2028-03-31 04:00:37.612130016 +0000 UTC" Jul 13 03:41:38.628913 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.628856 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15048h18m58.983278821s" Jul 13 03:41:38.747353 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.747318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:38.747825 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.747480 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:38.747961 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:38.747939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:38.748095 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:38.748057 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:39.749440 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:39.749405 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:39.749902 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:39.749555 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:40.329539 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:40.329505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:40.329539 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:40.329548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:40.329802 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.329667 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:40.329802 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.329716 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:41:44.329700282 +0000 UTC m=+9.105657961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:40.330142 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.330120 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:40.330236 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.330180 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:41:44.330163544 +0000 UTC m=+9.106121228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:40.429951 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:40.429906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:40.430183 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.430161 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:40.430280 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.430194 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:40.430280 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.430208 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:40.430280 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.430265 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:41:44.430246625 +0000 UTC m=+9.206204312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:40.746525 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:40.746489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:40.746696 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.746639 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:40.747113 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:40.747090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:40.747231 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:40.747188 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:41.746832 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:41.746793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:41.747350 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:41.746933 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:42.747341 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:42.747303 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:42.747811 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:42.747464 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:42.747968 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:42.747876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:42.748020 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:42.747974 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:43.746885 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:43.746847 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:43.747049 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:43.747000 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:44.362024 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:44.361969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:44.362460 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:44.362091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:44.362460 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.362123 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:44.362460 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.362202 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:41:52.362180165 +0000 UTC m=+17.138137847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:44.362460 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.362212 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:44.362460 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.362267 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:41:52.36224793 +0000 UTC m=+17.138205611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:44.462766 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:44.462727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:44.462937 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.462897 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:44.462937 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.462921 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:44.462937 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.462936 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:44.463120 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.463020 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:41:52.462998741 +0000 UTC m=+17.238956431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:44.746532 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:44.746450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:44.746699 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.746597 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:44.746972 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:44.746952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:44.747087 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:44.747066 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:45.748939 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:45.748563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:45.748939 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:45.748694 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:46.746362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:46.746281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:46.746538 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:46.746420 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:46.746647 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:46.746624 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:46.746773 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:46.746749 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:47.747099 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:47.747064 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:47.747541 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:47.747194 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:48.746788 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:48.746754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:48.746962 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:48.746754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:48.746962 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:48.746879 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:48.747094 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:48.746970 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:49.746499 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:49.746461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:49.746948 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:49.746591 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:50.746636 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:50.746595 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:50.747000 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:50.746705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:50.747000 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:50.746761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:50.747000 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:50.746844 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:51.747046 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:51.747010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:51.747434 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:51.747144 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:52.421116 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:52.421080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:52.421354 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:52.421136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:52.421354 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.421247 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:52.421354 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.421303 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:52.421354 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.421316 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:42:08.421294483 +0000 UTC m=+33.197252165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:41:52.421553 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.421359 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:42:08.421342783 +0000 UTC m=+33.197300461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:41:52.521990 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:52.521944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:52.522157 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.522106 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:41:52.522157 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.522130 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:41:52.522157 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.522147 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:52.522302 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.522219 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:42:08.522200764 +0000 UTC m=+33.298158454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:41:52.746540 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:52.746452 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:52.746540 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:52.746503 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:52.746742 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.746570 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:52.746742 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:52.746663 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:53.746438 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:53.746408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:53.746887 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:53.746535 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:54.747026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:54.746993 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:54.747434 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:54.747035 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:54.747434 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:54.747132 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:54.747434 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:54.747263 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:55.747211 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:55.746953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:55.747656 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:55.747306 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:56.746855 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:56.746822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:56.746855 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:56.746862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:56.747110 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:56.746955 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:56.747110 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:56.747072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:57.747235 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.747210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:57.747570 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:57.747334 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:57.910241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.910133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq9vg" event={"ID":"e22f94d2-04ca-470e-8664-5c33f1ab1691","Type":"ContainerStarted","Data":"20f3fe38d5c48c38ad7b70ea5e4c6c7e3493bcabd50f44bea4a8da4ce501b7b8"} Jul 13 03:41:57.911858 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.911824 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" event={"ID":"7aa2a3eea528d01b7f1c87c2d7e7e408","Type":"ContainerStarted","Data":"16bc5ecbd52988f00c2f0d486ad4cc4087845ff9376c198b41019825b4718a99"} Jul 13 03:41:57.913993 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.913959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" event={"ID":"611bfda6-ecf4-4294-bf2a-3b38e60dfde5","Type":"ContainerStarted","Data":"ef72c31abce80c7570af0d154e3674a68f47554451d062b0237a1a62747a8fea"} Jul 13 03:41:57.925889 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.925840 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cq9vg" podStartSLOduration=1.32155173 podStartE2EDuration="21.925824451s" totalImagesPullingTime="20.604272721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:57.925476157 +0000 UTC m=+22.701433864" watchObservedRunningTime="2026-07-13 03:41:57.925824451 +0000 UTC m=+22.701782151" Jul 13 03:41:57.943927 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.943889 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xpjhw" podStartSLOduration=1.404631233 podStartE2EDuration="21.943876226s" totalImagesPullingTime="20.539244993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:57.943720913 +0000 UTC m=+22.719678612" watchObservedRunningTime="2026-07-13 03:41:57.943876226 +0000 UTC m=+22.719833926" Jul 13 03:41:57.955484 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:57.955447 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" podStartSLOduration=20.955431228 podStartE2EDuration="20.955431228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:57.954964012 +0000 UTC m=+22.730921712" watchObservedRunningTime="2026-07-13 03:41:57.955431228 +0000 UTC m=+22.731388929" Jul 13 03:41:58.747071 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.746764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:41:58.747236 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.746809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:41:58.747236 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:58.747155 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:41:58.747757 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:58.747235 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:41:58.916962 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.916929 2562 generic.go:356] "Generic (PLEG): container finished" podID="84ea5db1b3c2451296d25e11b01000b7" containerID="0b6299bc7635bb4c3a0b317cbc700b36e50fe63e5a4d6f4b91817ac99e063516" exitCode=0 Jul 13 03:41:58.917118 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.917022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"84ea5db1b3c2451296d25e11b01000b7","Type":"ContainerDied","Data":"0b6299bc7635bb4c3a0b317cbc700b36e50fe63e5a4d6f4b91817ac99e063516"} Jul 13 03:41:58.918598 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.918574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" event={"ID":"103c32dd-423d-49a5-806f-11e8075c71ac","Type":"ContainerStarted","Data":"1281821f14c1ff4134a42c48dc39ed7c772c3d7fbed5473c71774a2b20afe829"} Jul 13 03:41:58.920014 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.919972 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfqj5" event={"ID":"c6e3f1a3-8dc2-4924-9eaf-c279e6940930","Type":"ContainerStarted","Data":"05cacc8431639dfee0e5e33e43e5de90174334ce096c7b0609d36574cd15715b"} Jul 13 03:41:58.921320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.921299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjvh9" event={"ID":"db23e086-5d21-4d81-9396-72be63d48a33","Type":"ContainerStarted","Data":"0e572ea3900a0acc4d44fa4d04b530f173bdb44da3ed6908dfce0d2e4d6b4872"} Jul 13 03:41:58.922680 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.922660 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="f85876d86fb7d333b8ace082071147c70a6abde6d0ec7bb08038c355678996a3" exitCode=0 Jul 13 03:41:58.922756 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.922720 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"f85876d86fb7d333b8ace082071147c70a6abde6d0ec7bb08038c355678996a3"} Jul 13 03:41:58.924281 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.924256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mg78z" event={"ID":"55459b3f-e007-4218-9cf8-85f8f7bb9976","Type":"ContainerStarted","Data":"ea5ecb4a7a1c6591b23492ee4f96b996f7da118d6eb29c87078200fa3a267110"} Jul 13 03:41:58.925767 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.925749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wpr6r" event={"ID":"e93eb64b-9ff7-4ada-ad80-286e166db78a","Type":"ContainerStarted","Data":"987b0b37391ac113372170190c5533bd0205fb32eb4ed359cb34b304dda21905"} Jul 13 03:41:58.928667 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.928651 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:41:58.928972 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.928950 2562 generic.go:356] "Generic (PLEG): container finished" podID="7cf560ba-63be-4f6d-a5f9-5af6100690d0" containerID="3fe2ab92de84f100d691d0330090dbb2268d6d1021d779d2802529b63cb7f53e" exitCode=1 Jul 13 03:41:58.929082 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"69287c8a4d30cd3deb6982baaebc32978e5072b1cea148fe4a8b6b8eddf9469e"} Jul 13 03:41:58.929082 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"6f124c17a146fe1114770039868112e53c61395ed3a80c9d1fe129ced424c4c4"} Jul 13 03:41:58.929082 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"6f372b836343ec80b05e6490e7fa3bf0287d3ef748c64f0eb7f4af0ffc9561cf"} Jul 13 03:41:58.929228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"f60e67ed8a61fa222e9e4e809a97d46951dab3213a6d835579ca2c0eed24a5c5"} Jul 13 03:41:58.929228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerDied","Data":"3fe2ab92de84f100d691d0330090dbb2268d6d1021d779d2802529b63cb7f53e"} Jul 13 03:41:58.929228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.929192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"52dfbe44739e6b04e73abfacfd5aa653d029008e6c66e286ea4a1991e4b50217"} Jul 13 03:41:58.941019 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.940968 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lfqj5" podStartSLOduration=2.406830618 podStartE2EDuration="22.940956178s" totalImagesPullingTime="20.53412556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:58.940943228 +0000 UTC m=+23.716900925" watchObservedRunningTime="2026-07-13 03:41:58.940956178 +0000 UTC m=+23.716913878" Jul 13 03:41:58.952642 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.952613 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wpr6r" podStartSLOduration=2.297475981 podStartE2EDuration="22.952602909s" totalImagesPullingTime="20.655126928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:58.95245044 +0000 UTC m=+23.728408137" watchObservedRunningTime="2026-07-13 03:41:58.952602909 +0000 UTC m=+23.728560608" Jul 13 03:41:58.965006 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.964920 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mg78z" podStartSLOduration=2.36077312 podStartE2EDuration="22.964905589s" totalImagesPullingTime="20.604132469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:58.964093798 +0000 UTC m=+23.740051530" watchObservedRunningTime="2026-07-13 03:41:58.964905589 +0000 UTC m=+23.740863290" Jul 13 03:41:58.996733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:58.996703 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tjvh9" podStartSLOduration=2.435496192 podStartE2EDuration="22.996691153s" totalImagesPullingTime="20.561194961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:58.996395394 +0000 UTC m=+23.772353093" watchObservedRunningTime="2026-07-13 03:41:58.996691153 +0000 UTC m=+23.772648852" Jul 13 03:41:59.747041 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:59.747013 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:41:59.747164 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:41:59.747146 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:41:59.884098 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:59.884077 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 03:41:59.933552 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:59.933519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"84ea5db1b3c2451296d25e11b01000b7","Type":"ContainerStarted","Data":"a9adb1bc35d260360342fbfc20ad59f76524a585282ca4fe5ac929d6f941d9e6"} Jul 13 03:41:59.935778 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:41:59.935751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" event={"ID":"103c32dd-423d-49a5-806f-11e8075c71ac","Type":"ContainerStarted","Data":"fcbc04ea6a795c1e71bb35e82243a258e960e1587d8b083cfa8c8b957dc7ada7"} Jul 13 03:42:00.678960 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.678839 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T03:41:59.884094894Z","UUID":"cf658709-64f1-449e-9ac6-32570dd0c919","Handler":null,"Name":"","Endpoint":""} Jul 13 03:42:00.680681 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.680651 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 03:42:00.680814 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.680695 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 03:42:00.746957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.746917 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:00.747123 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.746928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:00.747123 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:00.747068 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:00.747214 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:00.747195 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:00.941390 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.941326 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:42:00.941767 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:00.941685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"cfedc088b3ae11d52af1cdb675876678994c9e2df028de4310e5c4fe14100500"} Jul 13 03:42:01.747275 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:01.747243 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:01.747535 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:01.747370 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:01.946668 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:01.946636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" event={"ID":"103c32dd-423d-49a5-806f-11e8075c71ac","Type":"ContainerStarted","Data":"9e5d0968da1aefb1628535dfe76667e3f31c6f11eb825ec4d49124a547adb0fe"} Jul 13 03:42:01.966358 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:01.964358 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" podStartSLOduration=24.964339443 podStartE2EDuration="24.964339443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:41:59.95125053 +0000 UTC m=+24.727208230" watchObservedRunningTime="2026-07-13 03:42:01.964339443 +0000 UTC m=+26.740297144" Jul 13 03:42:01.966358 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:01.964546 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9jtm" podStartSLOduration=3.187095013 podStartE2EDuration="25.96453565s" totalImagesPullingTime="22.777440637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:01.963411467 +0000 UTC m=+26.739369166" watchObservedRunningTime="2026-07-13 03:42:01.96453565 +0000 UTC m=+26.740493351" Jul 13 03:42:02.257538 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.257508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:42:02.258239 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.258218 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:42:02.746849 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.746785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:02.747005 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:02.746893 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:02.747005 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.746958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:02.747118 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:02.747097 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:02.948605 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.948572 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:42:02.949017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:02.948930 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tjvh9" Jul 13 03:42:03.746952 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.746924 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:03.747144 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:03.747058 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:03.953208 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.953181 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:42:03.954107 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.953547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"71a64bba56b0364004b9ff25d59f62d85cf533a1d2138d2a6ecfe775a0441395"} Jul 13 03:42:03.954107 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.953766 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:03.954107 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.953944 2562 scope.go:117] "RemoveContainer" containerID="3fe2ab92de84f100d691d0330090dbb2268d6d1021d779d2802529b63cb7f53e" Jul 13 03:42:03.955609 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.955584 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="027de51d8cce04b5ed2052ac1de50f3934b62fad7caa90c72215438dfec62e26" exitCode=0 Jul 13 03:42:03.955719 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.955655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"027de51d8cce04b5ed2052ac1de50f3934b62fad7caa90c72215438dfec62e26"} Jul 13 03:42:03.970965 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:03.970947 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:04.746306 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.746270 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:04.746516 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:04.746405 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:04.746516 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.746462 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:04.746598 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:04.746577 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:04.961278 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.961252 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:42:04.961648 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.961593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" event={"ID":"7cf560ba-63be-4f6d-a5f9-5af6100690d0","Type":"ContainerStarted","Data":"e716d1cf3ee37ea00ac0dc4df3e47c72722d6dad86ef401049375efb06b408b2"} Jul 13 03:42:04.961724 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.961708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:04.961779 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.961770 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:04.977241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.977220 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:04.984952 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:04.984910 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" podStartSLOduration=8.103857329 podStartE2EDuration="28.98489578s" totalImagesPullingTime="20.881038451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:04.983920229 +0000 UTC m=+29.759877929" watchObservedRunningTime="2026-07-13 03:42:04.98489578 +0000 UTC m=+29.760853482" Jul 13 03:42:05.750231 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:05.748752 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:05.750231 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:05.748902 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:05.965204 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:05.965170 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="0671edbb8667928a30c3622990d87c80fddfb576036840945200a094cf2dba56" exitCode=0 Jul 13 03:42:05.965628 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:05.965257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"0671edbb8667928a30c3622990d87c80fddfb576036840945200a094cf2dba56"} Jul 13 03:42:06.747064 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:06.747035 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:06.747214 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:06.747135 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:06.747214 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:06.747195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:06.747299 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:06.747281 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:07.747145 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:07.747116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:07.747524 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:07.747204 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:07.971228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:07.971194 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="7d0c60e07dc2145cb61f2d04cc410ae98363e1e87714e228750f18d3f3afe552" exitCode=0 Jul 13 03:42:07.971228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:07.971233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"7d0c60e07dc2145cb61f2d04cc410ae98363e1e87714e228750f18d3f3afe552"} Jul 13 03:42:08.443385 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:08.443350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:08.443553 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:08.443418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:08.443553 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.443519 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 03:42:08.443553 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.443523 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:42:08.443671 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.443585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret podName:dbe98aea-97ba-462c-b92d-4d147e56fd4d nodeName:}" failed. No retries permitted until 2026-07-13 03:42:40.443568109 +0000 UTC m=+65.219525789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret") pod "global-pull-secret-syncer-45z9p" (UID: "dbe98aea-97ba-462c-b92d-4d147e56fd4d") : object "kube-system"/"original-pull-secret" not registered Jul 13 03:42:08.443671 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.443606 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs podName:446ecfd0-eaac-47cd-b3a8-6046826dcbac nodeName:}" failed. No retries permitted until 2026-07-13 03:42:40.443595754 +0000 UTC m=+65.219553437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs") pod "network-metrics-daemon-8ggwx" (UID: "446ecfd0-eaac-47cd-b3a8-6046826dcbac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 03:42:08.544023 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:08.543971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:08.544215 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.544182 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 03:42:08.544215 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.544212 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 03:42:08.544375 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.544227 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2c9xp for pod openshift-network-diagnostics/network-check-target-pcmzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:42:08.544375 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.544295 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp podName:06cbdc46-0047-4f05-a53a-a9285a2033bd nodeName:}" failed. No retries permitted until 2026-07-13 03:42:40.544274436 +0000 UTC m=+65.320232118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c9xp" (UniqueName: "kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp") pod "network-check-target-pcmzm" (UID: "06cbdc46-0047-4f05-a53a-a9285a2033bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 03:42:08.746610 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:08.746528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:08.746773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:08.746531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:08.746773 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.746679 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:08.746773 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:08.746724 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:09.749901 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:09.749868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:09.750361 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:09.749999 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:10.747280 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:10.747245 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:10.747466 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:10.747245 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:10.747466 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:10.747362 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:10.747575 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:10.747491 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:11.750054 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:11.750022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:11.750458 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:11.750133 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:12.746551 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:12.746522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:12.746769 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:12.746637 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:12.746769 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:12.746691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:12.746901 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:12.746767 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:12.985812 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:12.985779 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="fd2de9a0aeab806761f2d9baa8dbb2ff25056e14a15ffc0232d6f5f5db0a3faf" exitCode=0 Jul 13 03:42:12.986285 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:12.985835 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"fd2de9a0aeab806761f2d9baa8dbb2ff25056e14a15ffc0232d6f5f5db0a3faf"} Jul 13 03:42:13.747147 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:13.747117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:13.747316 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:13.747238 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:13.990255 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:13.990227 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd" containerID="414919cc0531f446c138adc6f0466db0647d5421edcbc2bc755621b93ec0f07d" exitCode=0 Jul 13 03:42:13.990619 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:13.990262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerDied","Data":"414919cc0531f446c138adc6f0466db0647d5421edcbc2bc755621b93ec0f07d"} Jul 13 03:42:14.746714 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:14.746684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:14.746886 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:14.746775 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:14.746886 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:14.746862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:14.747029 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:14.746960 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:14.995359 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:14.995330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" event={"ID":"4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd","Type":"ContainerStarted","Data":"dad923927832500ece8652f413724c4423a547bd6b026228c0cdc6e6c18022ee"} Jul 13 03:42:15.015220 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:15.015170 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jdnr6" podStartSLOduration=8.097390273 podStartE2EDuration="39.015157432s" totalImagesPullingTime="30.917767159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:15.014938659 +0000 UTC m=+39.790896358" watchObservedRunningTime="2026-07-13 03:42:15.015157432 +0000 UTC m=+39.791115132" Jul 13 03:42:15.747736 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:15.747707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:15.747893 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:15.747785 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:16.746991 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:16.746942 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:16.746991 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:16.746991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:16.747399 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:16.747087 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:16.747399 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:16.747203 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:17.749396 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:17.749368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:17.749776 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:17.749461 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:18.746790 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:18.746758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:18.746961 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:18.746762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:18.746961 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:18.746852 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:18.746961 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:18.746938 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:19.746586 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:19.746552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:19.746940 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:19.746678 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:20.746409 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:20.746380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:20.746571 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:20.746388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:20.746571 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:20.746468 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:20.746571 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:20.746557 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:21.747056 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:21.747026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:21.747562 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:21.747120 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:22.746796 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:22.746760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:22.747001 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:22.746760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:22.747001 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:22.746864 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:22.747001 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:22.746932 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:23.747177 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:23.747008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:23.747523 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:23.747262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:24.188709 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.188675 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-45z9p"] Jul 13 03:42:24.188865 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.188776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:24.188910 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:24.188858 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:24.192508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.192472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pcmzm"] Jul 13 03:42:24.192639 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.192582 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:24.192704 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:24.192671 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:24.193213 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.193191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ggwx"] Jul 13 03:42:24.193306 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:24.193279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:24.193396 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:24.193376 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:25.747970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:25.747937 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:25.748520 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:25.748037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:25.748520 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:25.748069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:25.748520 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:25.748077 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:25.748520 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:25.748144 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:25.748520 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:25.748244 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:27.747268 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:27.747236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:27.747651 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:27.747236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:27.747651 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:27.747361 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ggwx" podUID="446ecfd0-eaac-47cd-b3a8-6046826dcbac" Jul 13 03:42:27.747651 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:27.747377 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:27.747651 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:27.747446 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pcmzm" podUID="06cbdc46-0047-4f05-a53a-a9285a2033bd" Jul 13 03:42:27.747651 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:27.747491 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45z9p" podUID="dbe98aea-97ba-462c-b92d-4d147e56fd4d" Jul 13 03:42:28.539249 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.539221 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeReady" Jul 13 03:42:28.539416 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.539337 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 03:42:28.583029 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.582992 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-54vc8"] Jul 13 03:42:28.602902 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.602877 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8v5qz"] Jul 13 03:42:28.603040 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.603008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.604827 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.604804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gjttq\"" Jul 13 03:42:28.604827 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.604835 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 03:42:28.605131 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.605114 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 03:42:28.622958 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.622932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8v5qz"] Jul 13 03:42:28.622958 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.622961 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54vc8"] Jul 13 03:42:28.623113 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.623057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.625191 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.625176 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 03:42:28.625284 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.625213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 03:42:28.625452 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.625436 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 03:42:28.625498 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.625444 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tj96r\"" Jul 13 03:42:28.682997 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.682924 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dcc4t"] Jul 13 03:42:28.693313 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643c3520-aed7-426c-861c-82bb1830bebf-metrics-tls\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.693454 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643c3520-aed7-426c-861c-82bb1830bebf-tmp-dir\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.693454 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643c3520-aed7-426c-861c-82bb1830bebf-config-volume\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.693454 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h8wt\" (UniqueName: \"kubernetes.io/projected/643c3520-aed7-426c-861c-82bb1830bebf-kube-api-access-7h8wt\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.693625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hzg\" (UniqueName: \"kubernetes.io/projected/c1b63269-4ba5-4d49-9026-74118fbc5c23-kube-api-access-r4hzg\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.693625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.693490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b63269-4ba5-4d49-9026-74118fbc5c23-cert\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.704279 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.704255 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dcc4t"] Jul 13 03:42:28.704377 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.704363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.706263 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.706248 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 03:42:28.706522 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.706508 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 03:42:28.706582 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.706510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 03:42:28.706813 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.706800 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5zfx\"" Jul 13 03:42:28.706857 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.706834 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 03:42:28.793867 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.793836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.793893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643c3520-aed7-426c-861c-82bb1830bebf-config-volume\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.793922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8wt\" (UniqueName: \"kubernetes.io/projected/643c3520-aed7-426c-861c-82bb1830bebf-kube-api-access-7h8wt\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.793947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hzg\" (UniqueName: \"kubernetes.io/projected/c1b63269-4ba5-4d49-9026-74118fbc5c23-kube-api-access-r4hzg\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.793971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-crio-socket\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794019 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-data-volume\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b63269-4ba5-4d49-9026-74118fbc5c23-cert\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643c3520-aed7-426c-861c-82bb1830bebf-metrics-tls\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643c3520-aed7-426c-861c-82bb1830bebf-tmp-dir\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.794404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxkq\" (UniqueName: \"kubernetes.io/projected/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-api-access-5xxkq\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.794739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643c3520-aed7-426c-861c-82bb1830bebf-tmp-dir\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.794739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.794624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643c3520-aed7-426c-861c-82bb1830bebf-config-volume\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.797934 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.797915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643c3520-aed7-426c-861c-82bb1830bebf-metrics-tls\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.798058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.798040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1b63269-4ba5-4d49-9026-74118fbc5c23-cert\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.801334 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.801314 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hzg\" (UniqueName: \"kubernetes.io/projected/c1b63269-4ba5-4d49-9026-74118fbc5c23-kube-api-access-r4hzg\") pod \"ingress-canary-8v5qz\" (UID: \"c1b63269-4ba5-4d49-9026-74118fbc5c23\") " pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:28.801427 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.801347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8wt\" (UniqueName: \"kubernetes.io/projected/643c3520-aed7-426c-861c-82bb1830bebf-kube-api-access-7h8wt\") pod \"dns-default-54vc8\" (UID: \"643c3520-aed7-426c-861c-82bb1830bebf\") " pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.895306 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-crio-socket\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895306 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-data-volume\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895472 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895510 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-crio-socket\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895585 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-data-volume\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895632 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxkq\" (UniqueName: \"kubernetes.io/projected/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-api-access-5xxkq\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895665 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.895914 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.895891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.897751 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.897730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.902706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.902681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxkq\" (UniqueName: \"kubernetes.io/projected/bd24fd38-323b-4b4d-89cd-48a3a99bbfe6-kube-api-access-5xxkq\") pod \"insights-runtime-extractor-dcc4t\" (UID: \"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6\") " pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:28.912533 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.912502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:28.930618 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:28.930602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8v5qz" Jul 13 03:42:29.015526 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.015348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dcc4t" Jul 13 03:42:29.111189 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.111158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8v5qz"] Jul 13 03:42:29.114414 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:29.114385 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b63269_4ba5_4d49_9026_74118fbc5c23.slice/crio-f257af485c848c6f3a04a10475027ad78adf038bd1a375851a7d253b81320501 WatchSource:0}: Error finding container f257af485c848c6f3a04a10475027ad78adf038bd1a375851a7d253b81320501: Status 404 returned error can't find the container with id f257af485c848c6f3a04a10475027ad78adf038bd1a375851a7d253b81320501 Jul 13 03:42:29.116091 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.116066 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54vc8"] Jul 13 03:42:29.119575 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:29.119543 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643c3520_aed7_426c_861c_82bb1830bebf.slice/crio-02c58cf0b1918ce5e3f9a49f7cd41dbd512f75f859b71215ef4f8cf08fa8aeca WatchSource:0}: Error finding container 02c58cf0b1918ce5e3f9a49f7cd41dbd512f75f859b71215ef4f8cf08fa8aeca: Status 404 returned error can't find the container with id 02c58cf0b1918ce5e3f9a49f7cd41dbd512f75f859b71215ef4f8cf08fa8aeca Jul 13 03:42:29.167509 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.167478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dcc4t"] Jul 13 03:42:29.170551 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:29.170521 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd24fd38_323b_4b4d_89cd_48a3a99bbfe6.slice/crio-8f8e391c9653fad5ef71744e076e21e8bad883488a5ed11e1910d961f7c4b6b4 WatchSource:0}: Error finding container 8f8e391c9653fad5ef71744e076e21e8bad883488a5ed11e1910d961f7c4b6b4: Status 404 returned error can't find the container with id 8f8e391c9653fad5ef71744e076e21e8bad883488a5ed11e1910d961f7c4b6b4 Jul 13 03:42:29.747653 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.747619 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:29.747938 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.747663 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:29.747938 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.747803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:29.749674 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.749651 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vc8qt\"" Jul 13 03:42:29.749876 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.749859 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 03:42:29.750446 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.750426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 03:42:29.750640 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.750624 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 03:42:29.750807 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.750743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 03:42:29.750878 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:29.750869 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qx94t\"" Jul 13 03:42:30.031776 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.031672 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8v5qz" event={"ID":"c1b63269-4ba5-4d49-9026-74118fbc5c23","Type":"ContainerStarted","Data":"f257af485c848c6f3a04a10475027ad78adf038bd1a375851a7d253b81320501"} Jul 13 03:42:30.034331 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.034261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54vc8" event={"ID":"643c3520-aed7-426c-861c-82bb1830bebf","Type":"ContainerStarted","Data":"02c58cf0b1918ce5e3f9a49f7cd41dbd512f75f859b71215ef4f8cf08fa8aeca"} Jul 13 03:42:30.036716 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.036641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcc4t" event={"ID":"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6","Type":"ContainerStarted","Data":"8f8e391c9653fad5ef71744e076e21e8bad883488a5ed11e1910d961f7c4b6b4"} Jul 13 03:42:30.279131 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.278007 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6xltc"] Jul 13 03:42:30.288925 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.287478 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.292442 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.292409 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 03:42:30.292753 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.292734 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 03:42:30.292839 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.292799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 03:42:30.292839 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.292819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t4p2n\"" Jul 13 03:42:30.292970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.292940 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 13 03:42:30.293107 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.293081 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 13 03:42:30.293553 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.293471 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6xltc"] Jul 13 03:42:30.307394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.307086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.307394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.307179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c26d586b-9e78-4739-9b47-7600717c3298-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.307394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.307235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq2f\" (UniqueName: \"kubernetes.io/projected/c26d586b-9e78-4739-9b47-7600717c3298-kube-api-access-kkq2f\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.307394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.307283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.408237 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.408173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.408391 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.408244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.408391 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.408292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c26d586b-9e78-4739-9b47-7600717c3298-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.408391 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.408344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq2f\" (UniqueName: \"kubernetes.io/projected/c26d586b-9e78-4739-9b47-7600717c3298-kube-api-access-kkq2f\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.408752 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:30.408731 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 13 03:42:30.408853 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:30.408798 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls podName:c26d586b-9e78-4739-9b47-7600717c3298 nodeName:}" failed. No retries permitted until 2026-07-13 03:42:30.908778681 +0000 UTC m=+55.684736364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls") pod "prometheus-operator-64785b6675-6xltc" (UID: "c26d586b-9e78-4739-9b47-7600717c3298") : secret "prometheus-operator-tls" not found Jul 13 03:42:30.409524 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.409497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c26d586b-9e78-4739-9b47-7600717c3298-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.423243 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.423193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.423422 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.423399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq2f\" (UniqueName: \"kubernetes.io/projected/c26d586b-9e78-4739-9b47-7600717c3298-kube-api-access-kkq2f\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.913134 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.913092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:30.916368 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:30.916319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26d586b-9e78-4739-9b47-7600717c3298-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6xltc\" (UID: \"c26d586b-9e78-4739-9b47-7600717c3298\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:31.205685 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.205382 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" Jul 13 03:42:31.348792 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.348758 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:42:31.360422 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.360392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.362303 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.362277 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:42:31.364600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.363447 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 03:42:31.364600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.363591 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 03:42:31.364600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.363721 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6nlsh\"" Jul 13 03:42:31.364600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.364452 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 03:42:31.364600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.364594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 03:42:31.367241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.365225 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 03:42:31.367241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.365784 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 03:42:31.367241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.366111 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 03:42:31.416845 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.416809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.416845 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.416842 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.417078 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.416916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.417078 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.417002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.417078 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.417031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc76k\" (UniqueName: \"kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.417078 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.417048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517476 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517657 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517657 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517657 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517657 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.517882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.517681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc76k\" (UniqueName: \"kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.518538 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.518249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.518538 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.518312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.518722 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.518625 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.520755 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.520728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.520873 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.520844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.528840 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.528816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc76k\" (UniqueName: \"kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k\") pod \"console-7b56f77d5f-xj6sr\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:31.676671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:31.676633 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:35.031177 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.031148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6xltc"] Jul 13 03:42:35.036562 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:35.036532 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26d586b_9e78_4739_9b47_7600717c3298.slice/crio-55671cf11060bdc8fff2cda2ba9ad2e3bc85d6e7081e5a03db956ff22e464dd3 WatchSource:0}: Error finding container 55671cf11060bdc8fff2cda2ba9ad2e3bc85d6e7081e5a03db956ff22e464dd3: Status 404 returned error can't find the container with id 55671cf11060bdc8fff2cda2ba9ad2e3bc85d6e7081e5a03db956ff22e464dd3 Jul 13 03:42:35.053319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.053287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" event={"ID":"c26d586b-9e78-4739-9b47-7600717c3298","Type":"ContainerStarted","Data":"55671cf11060bdc8fff2cda2ba9ad2e3bc85d6e7081e5a03db956ff22e464dd3"} Jul 13 03:42:35.055439 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.055411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcc4t" event={"ID":"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6","Type":"ContainerStarted","Data":"ef262a6e7425778549392e6d61e54e1c9be0a09f44f80201709c610f553fa0f5"} Jul 13 03:42:35.058754 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.058728 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:42:35.060655 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.060596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54vc8" event={"ID":"643c3520-aed7-426c-861c-82bb1830bebf","Type":"ContainerStarted","Data":"ea1e608eb9de1e6377d027fc28ec4a619e3b09665595e603de75e0d22ebc68ee"} Jul 13 03:42:35.064758 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.064512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8v5qz" event={"ID":"c1b63269-4ba5-4d49-9026-74118fbc5c23","Type":"ContainerStarted","Data":"c8d8c328cea1a234ffea955e5f6e456af63effb2e96ea896c6a74b1eadc1fbee"} Jul 13 03:42:35.065319 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:35.065083 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ae9d45_0037_4870_9daa_326801a8afea.slice/crio-78d996a799a622eb8c827521b9a96a144c92c5ec18f143fae0c60e04d6c90154 WatchSource:0}: Error finding container 78d996a799a622eb8c827521b9a96a144c92c5ec18f143fae0c60e04d6c90154: Status 404 returned error can't find the container with id 78d996a799a622eb8c827521b9a96a144c92c5ec18f143fae0c60e04d6c90154 Jul 13 03:42:35.087271 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:35.087230 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8v5qz" podStartSLOduration=1.301320207 podStartE2EDuration="7.087216167s" totalImagesPullingTime="5.78589596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:35.085958351 +0000 UTC m=+59.861916069" watchObservedRunningTime="2026-07-13 03:42:35.087216167 +0000 UTC m=+59.863173867" Jul 13 03:42:36.070835 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.070786 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56f77d5f-xj6sr" event={"ID":"63ae9d45-0037-4870-9daa-326801a8afea","Type":"ContainerStarted","Data":"78d996a799a622eb8c827521b9a96a144c92c5ec18f143fae0c60e04d6c90154"} Jul 13 03:42:36.073575 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.073487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcc4t" event={"ID":"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6","Type":"ContainerStarted","Data":"f84f28298a798fbd4b49095bfbd3a74dbb2fe9cbe78f0cf81bce34d279b2dad1"} Jul 13 03:42:36.076669 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.076639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54vc8" event={"ID":"643c3520-aed7-426c-861c-82bb1830bebf","Type":"ContainerStarted","Data":"05546b2c99e09288e8b710e6bbaeee0673533a5715da96b6066955cd0b31107e"} Jul 13 03:42:36.076797 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.076705 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:36.098193 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.098129 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-54vc8" podStartSLOduration=2.351637816 podStartE2EDuration="8.098111616s" totalImagesPullingTime="5.7464738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:36.091426459 +0000 UTC m=+60.867384161" watchObservedRunningTime="2026-07-13 03:42:36.098111616 +0000 UTC m=+60.874069318" Jul 13 03:42:36.979937 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:36.979863 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5xf7" Jul 13 03:42:37.083382 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:37.083341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" event={"ID":"c26d586b-9e78-4739-9b47-7600717c3298","Type":"ContainerStarted","Data":"a693eea37691c3ee567bc821b679d53e246904604b33441105c37b29de5126dd"} Jul 13 03:42:37.083822 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:37.083401 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" event={"ID":"c26d586b-9e78-4739-9b47-7600717c3298","Type":"ContainerStarted","Data":"b7bf3b350893a5e737ca661bef65f23ac1975cc7a95391baea14595893aca047"} Jul 13 03:42:37.098002 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:37.097938 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-6xltc" podStartSLOduration=5.461098129 podStartE2EDuration="7.097919622s" totalImagesPullingTime="1.636821493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:37.097671068 +0000 UTC m=+61.873628793" watchObservedRunningTime="2026-07-13 03:42:37.097919622 +0000 UTC m=+61.873877322" Jul 13 03:42:38.627808 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.627766 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5"] Jul 13 03:42:38.630097 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.630069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.630527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.630500 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xr84f"] Jul 13 03:42:38.632051 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.632021 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 13 03:42:38.632243 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.632222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 13 03:42:38.633254 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.632819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c9s7b\"" Jul 13 03:42:38.633386 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.633310 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.635667 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.635645 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 13 03:42:38.636559 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.635938 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 13 03:42:38.636559 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.636167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7sz7q\"" Jul 13 03:42:38.636559 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.636253 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 13 03:42:38.639876 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.639857 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5"] Jul 13 03:42:38.656124 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.656103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xr84f"] Jul 13 03:42:38.657693 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.657660 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mjb5t"] Jul 13 03:42:38.660145 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.660127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.662586 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.662407 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjmmw\"" Jul 13 03:42:38.662586 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.662432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 03:42:38.662739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.662656 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 03:42:38.662855 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.662825 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 03:42:38.675283 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9f708cc-5ab0-4823-b46f-a34e9c739683-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.675394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.675394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.675394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.675394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.675612 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.675612 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.675612 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrtp\" (UniqueName: \"kubernetes.io/projected/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-kube-api-access-qwrtp\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.675757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpn9w\" (UniqueName: \"kubernetes.io/projected/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-api-access-tpn9w\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.675757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.675658 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776356 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zb9\" (UniqueName: \"kubernetes.io/projected/443e8e18-faf1-409b-9461-49bc5e1445f5-kube-api-access-z4zb9\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpn9w\" (UniqueName: \"kubernetes.io/projected/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-api-access-tpn9w\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9f708cc-5ab0-4823-b46f-a34e9c739683-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-sys\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrtp\" (UniqueName: \"kubernetes.io/projected/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-kube-api-access-qwrtp\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.776773 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-wtmp\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776786 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-metrics-client-ca\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-root\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-textfile\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9f708cc-5ab0-4823-b46f-a34e9c739683-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.776942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.777153 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:38.777124 2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 13 03:42:38.777480 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:38.777196 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls podName:ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1 nodeName:}" failed. No retries permitted until 2026-07-13 03:42:39.277177296 +0000 UTC m=+64.053134974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-q2jt5" (UID: "ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1") : secret "openshift-state-metrics-tls" not found Jul 13 03:42:38.777480 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.777351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.777480 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.777428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.777660 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.777634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f708cc-5ab0-4823-b46f-a34e9c739683-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.779558 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.779513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.779558 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.779551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.779716 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.779578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.786836 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.786807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrtp\" (UniqueName: \"kubernetes.io/projected/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-kube-api-access-qwrtp\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:38.787844 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.787821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpn9w\" (UniqueName: \"kubernetes.io/projected/b9f708cc-5ab0-4823-b46f-a34e9c739683-kube-api-access-tpn9w\") pod \"kube-state-metrics-785778ff74-xr84f\" (UID: \"b9f708cc-5ab0-4823-b46f-a34e9c739683\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:38.878238 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878238 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zb9\" (UniqueName: \"kubernetes.io/projected/443e8e18-faf1-409b-9461-49bc5e1445f5-kube-api-access-z4zb9\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:38.878322 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-sys\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:38.878409 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls podName:443e8e18-faf1-409b-9461-49bc5e1445f5 nodeName:}" failed. No retries permitted until 2026-07-13 03:42:39.378386236 +0000 UTC m=+64.154343931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls") pod "node-exporter-mjb5t" (UID: "443e8e18-faf1-409b-9461-49bc5e1445f5") : secret "node-exporter-tls" not found Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878435 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-sys\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-wtmp\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-metrics-client-ca\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-root\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-textfile\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-wtmp\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e8e18-faf1-409b-9461-49bc5e1445f5-root\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.878970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.878929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-textfile\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.879275 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.879077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e8e18-faf1-409b-9461-49bc5e1445f5-metrics-client-ca\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.881349 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.881305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.890896 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.890870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zb9\" (UniqueName: \"kubernetes.io/projected/443e8e18-faf1-409b-9461-49bc5e1445f5-kube-api-access-z4zb9\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:38.953127 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:38.953091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" Jul 13 03:42:39.282037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.282010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:39.284359 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.284336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q2jt5\" (UID: \"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:39.383104 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.383074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:39.385153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.385136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e8e18-faf1-409b-9461-49bc5e1445f5-node-exporter-tls\") pod \"node-exporter-mjb5t\" (UID: \"443e8e18-faf1-409b-9461-49bc5e1445f5\") " pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:39.389927 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.389898 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xr84f"] Jul 13 03:42:39.394071 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:39.394046 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f708cc_5ab0_4823_b46f_a34e9c739683.slice/crio-045b241381f99bd079a68367d926d5baab3909294eb45444708a30092c99c6a3 WatchSource:0}: Error finding container 045b241381f99bd079a68367d926d5baab3909294eb45444708a30092c99c6a3: Status 404 returned error can't find the container with id 045b241381f99bd079a68367d926d5baab3909294eb45444708a30092c99c6a3 Jul 13 03:42:39.546638 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.546580 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" Jul 13 03:42:39.569935 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.569906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjb5t" Jul 13 03:42:39.579084 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:39.579055 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443e8e18_faf1_409b_9461_49bc5e1445f5.slice/crio-6b5c0683869cde6fdacac33997fd88e7901318cc33f82dfe95f1e633433b78b7 WatchSource:0}: Error finding container 6b5c0683869cde6fdacac33997fd88e7901318cc33f82dfe95f1e633433b78b7: Status 404 returned error can't find the container with id 6b5c0683869cde6fdacac33997fd88e7901318cc33f82dfe95f1e633433b78b7 Jul 13 03:42:39.670417 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.670364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5"] Jul 13 03:42:39.673091 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:39.673063 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef39cb7f_a940_4ce5_ab54_bdcbe47ab4f1.slice/crio-aaf9454fd7456eb9c9bf3fc9e25c127f8f2ea6d3fd4455aa24d602a0d3df9498 WatchSource:0}: Error finding container aaf9454fd7456eb9c9bf3fc9e25c127f8f2ea6d3fd4455aa24d602a0d3df9498: Status 404 returned error can't find the container with id aaf9454fd7456eb9c9bf3fc9e25c127f8f2ea6d3fd4455aa24d602a0d3df9498 Jul 13 03:42:39.720701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.720668 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:42:39.725664 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.725644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.727475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.727458 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 13 03:42:39.728026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.727616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kbkqg\"" Jul 13 03:42:39.728026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.727842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 13 03:42:39.728026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.727888 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 13 03:42:39.728026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.727951 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 13 03:42:39.728313 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.728112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 13 03:42:39.728888 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.728622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 13 03:42:39.728888 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.728645 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 13 03:42:39.728888 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.728658 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 13 03:42:39.728888 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.728690 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 13 03:42:39.741077 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.739074 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:42:39.752625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.752397 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:42:39.755057 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.755038 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.762282 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.762259 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 03:42:39.763965 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.763943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:42:39.785662 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.785783 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.785783 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.785783 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.785783 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67j5\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.786017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.785945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6vx\" (UniqueName: \"kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.786329 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786329 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786329 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.786471 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786471 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.786471 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786471 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786746 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786746 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.786746 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.786551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.887629 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.887629 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.887881 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.887881 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888032 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887872 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6vx\" (UniqueName: \"kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.888032 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.887957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888032 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.888202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889062 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.888382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67j5\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.889496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.889776 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.889751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.890664 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:39.889963 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle podName:7dd5e715-acf8-4824-bbf4-6a66fcf9c54c nodeName:}" failed. No retries permitted until 2026-07-13 03:42:40.389942963 +0000 UTC m=+65.165900659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c") : configmap references non-existent config key: ca-bundle.crt Jul 13 03:42:39.890664 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.890485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.890664 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.890590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.891041 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.891011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.891470 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.891433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.891660 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.891637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.892569 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.892541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.892989 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.892948 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.893235 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.893211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.893571 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.893545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:39.894349 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.894349 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894310 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.894662 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.894739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.894739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894721 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.894846 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.894766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.897940 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.897918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67j5\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:39.898361 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:39.898334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6vx\" (UniqueName: \"kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx\") pod \"console-644df44c57-szpfs\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:40.077357 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.077319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:40.098498 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.098464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" event={"ID":"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1","Type":"ContainerStarted","Data":"66a6fc7e8fbc47538ca19ef616357191b7535a16afa91da2d1b80d6676ba2243"} Jul 13 03:42:40.098629 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.098529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" event={"ID":"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1","Type":"ContainerStarted","Data":"6ecaa96bc39c16a94ba9ad56ea96ec76a8438ef30811a459cd7ee635acb0ab23"} Jul 13 03:42:40.098629 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.098549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" event={"ID":"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1","Type":"ContainerStarted","Data":"aaf9454fd7456eb9c9bf3fc9e25c127f8f2ea6d3fd4455aa24d602a0d3df9498"} Jul 13 03:42:40.099774 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.099741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" event={"ID":"b9f708cc-5ab0-4823-b46f-a34e9c739683","Type":"ContainerStarted","Data":"045b241381f99bd079a68367d926d5baab3909294eb45444708a30092c99c6a3"} Jul 13 03:42:40.101526 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.101497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56f77d5f-xj6sr" event={"ID":"63ae9d45-0037-4870-9daa-326801a8afea","Type":"ContainerStarted","Data":"452b1ff52da31b8bea1a0b88f046b9e4913152c41b40e3e04fe1e39acf08162c"} Jul 13 03:42:40.103165 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.103126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5t" event={"ID":"443e8e18-faf1-409b-9461-49bc5e1445f5","Type":"ContainerStarted","Data":"6b5c0683869cde6fdacac33997fd88e7901318cc33f82dfe95f1e633433b78b7"} Jul 13 03:42:40.105884 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.105859 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcc4t" event={"ID":"bd24fd38-323b-4b4d-89cd-48a3a99bbfe6","Type":"ContainerStarted","Data":"938d0e22567114f8e1f7ae77628133ee18a57b02dd4af153c960b47e813825dc"} Jul 13 03:42:40.116186 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.116127 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b56f77d5f-xj6sr" podStartSLOduration=4.913726378 podStartE2EDuration="9.116108258s" totalImagesPullingTime="4.20238188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:40.11572438 +0000 UTC m=+64.891682080" watchObservedRunningTime="2026-07-13 03:42:40.116108258 +0000 UTC m=+64.892065959" Jul 13 03:42:40.130972 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.130931 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dcc4t" podStartSLOduration=2.193725853 podStartE2EDuration="12.13091688s" totalImagesPullingTime="9.937191027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:42:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:40.130036714 +0000 UTC m=+64.905994416" watchObservedRunningTime="2026-07-13 03:42:40.13091688 +0000 UTC m=+64.906874580" Jul 13 03:42:40.231849 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.231782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:42:40.394008 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.393908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:40.396754 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.396369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:40.494516 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.494480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:40.494673 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.494528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:40.496428 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.496379 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 03:42:40.496570 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.496552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 03:42:40.507471 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.507448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbe98aea-97ba-462c-b92d-4d147e56fd4d-original-pull-secret\") pod \"global-pull-secret-syncer-45z9p\" (UID: \"dbe98aea-97ba-462c-b92d-4d147e56fd4d\") " pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:40.507595 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.507495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/446ecfd0-eaac-47cd-b3a8-6046826dcbac-metrics-certs\") pod \"network-metrics-daemon-8ggwx\" (UID: \"446ecfd0-eaac-47cd-b3a8-6046826dcbac\") " pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:40.574225 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.574193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45z9p" Jul 13 03:42:40.583324 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.583303 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qx94t\"" Jul 13 03:42:40.592608 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.592584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ggwx" Jul 13 03:42:40.595609 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.595581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:40.597446 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.597428 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 03:42:40.608317 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.608293 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 03:42:40.620222 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.620199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9xp\" (UniqueName: \"kubernetes.io/projected/06cbdc46-0047-4f05-a53a-a9285a2033bd-kube-api-access-2c9xp\") pod \"network-check-target-pcmzm\" (UID: \"06cbdc46-0047-4f05-a53a-a9285a2033bd\") " pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:40.637148 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.637122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:42:40.639202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.639100 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm"] Jul 13 03:42:40.641835 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.641817 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.644084 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.643869 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 13 03:42:40.644084 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.643874 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 13 03:42:40.644084 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.643881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b4pqgu9vjogqn\"" Jul 13 03:42:40.644084 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.644005 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gxwh\"" Jul 13 03:42:40.644323 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.644272 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 13 03:42:40.644550 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.644523 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 13 03:42:40.644637 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.644551 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 13 03:42:40.657564 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.657545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm"] Jul 13 03:42:40.696765 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-grpc-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-metrics-client-ca\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.696948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dv8\" (UniqueName: \"kubernetes.io/projected/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-kube-api-access-t5dv8\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.697210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.697016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798364 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-grpc-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798376 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-metrics-client-ca\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dv8\" (UniqueName: \"kubernetes.io/projected/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-kube-api-access-t5dv8\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.798762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.798580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.799899 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.799507 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-metrics-client-ca\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.801452 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.801408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.802070 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.802040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.802449 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.802428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.802548 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.802530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.802614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.802595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-thanos-querier-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.802772 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.802754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-secret-grpc-tls\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.806617 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.806596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dv8\" (UniqueName: \"kubernetes.io/projected/babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e-kube-api-access-t5dv8\") pod \"thanos-querier-b7bbf66d4-tcqnm\" (UID: \"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e\") " pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:40.866372 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.866347 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vc8qt\"" Jul 13 03:42:40.874882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.874841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:40.953272 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:40.953194 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:41.113715 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.113681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644df44c57-szpfs" event={"ID":"2611ecbe-2ee9-4fe5-92ee-06885230266a","Type":"ContainerStarted","Data":"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128"} Jul 13 03:42:41.113819 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.113738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644df44c57-szpfs" event={"ID":"2611ecbe-2ee9-4fe5-92ee-06885230266a","Type":"ContainerStarted","Data":"5ede59c5749b06c87bdec81d3766b94f2431d904255796ce6a2f3807ef773324"} Jul 13 03:42:41.143100 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.142754 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-644df44c57-szpfs" podStartSLOduration=2.142730342 podStartE2EDuration="2.142730342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:41.140830053 +0000 UTC m=+65.916787879" watchObservedRunningTime="2026-07-13 03:42:41.142730342 +0000 UTC m=+65.918688097" Jul 13 03:42:41.316428 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.315825 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-45z9p"] Jul 13 03:42:41.330312 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.330238 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pcmzm"] Jul 13 03:42:41.340399 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:41.340052 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe98aea_97ba_462c_b92d_4d147e56fd4d.slice/crio-46f083b6e1a67da5b0500a86e52449b4dbe5d9bcd9cf2e1032354b1118d40ce2 WatchSource:0}: Error finding container 46f083b6e1a67da5b0500a86e52449b4dbe5d9bcd9cf2e1032354b1118d40ce2: Status 404 returned error can't find the container with id 46f083b6e1a67da5b0500a86e52449b4dbe5d9bcd9cf2e1032354b1118d40ce2 Jul 13 03:42:41.342725 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.342652 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:42:41.344332 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:41.344285 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd5e715_acf8_4824_bbf4_6a66fcf9c54c.slice/crio-842c1dc849c275748580c4cfde97b2a05866a44e3fad8dd0159dea8337e9392a WatchSource:0}: Error finding container 842c1dc849c275748580c4cfde97b2a05866a44e3fad8dd0159dea8337e9392a: Status 404 returned error can't find the container with id 842c1dc849c275748580c4cfde97b2a05866a44e3fad8dd0159dea8337e9392a Jul 13 03:42:41.563489 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.563457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ggwx"] Jul 13 03:42:41.566820 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:41.566798 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446ecfd0_eaac_47cd_b3a8_6046826dcbac.slice/crio-bbd621d77ffe0e81c263c8e4bc4b42dfafca5f0eefe134a92eb0b6dfde41aec6 WatchSource:0}: Error finding container bbd621d77ffe0e81c263c8e4bc4b42dfafca5f0eefe134a92eb0b6dfde41aec6: Status 404 returned error can't find the container with id bbd621d77ffe0e81c263c8e4bc4b42dfafca5f0eefe134a92eb0b6dfde41aec6 Jul 13 03:42:41.571308 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.571281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm"] Jul 13 03:42:41.575734 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:41.575714 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabf3c3d_8784_47f4_9aa2_ffe8b3f3a40e.slice/crio-c6c8b399a93d14a98da58be7c00bbd905fad3788eaeb2f4f3fb0d7625a06d312 WatchSource:0}: Error finding container c6c8b399a93d14a98da58be7c00bbd905fad3788eaeb2f4f3fb0d7625a06d312: Status 404 returned error can't find the container with id c6c8b399a93d14a98da58be7c00bbd905fad3788eaeb2f4f3fb0d7625a06d312 Jul 13 03:42:41.677175 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.677130 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:41.677359 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.677200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:42:41.678536 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.678498 2562 patch_prober.go:28] interesting pod/console-7b56f77d5f-xj6sr container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 13 03:42:41.678651 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:41.678555 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7b56f77d5f-xj6sr" podUID="63ae9d45-0037-4870-9daa-326801a8afea" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 13 03:42:42.140846 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.139913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" event={"ID":"ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1","Type":"ContainerStarted","Data":"d0b27a03732904b71ed0c231701beb8af8a046de5b5f0bffa63ea8dc90964f6b"} Jul 13 03:42:42.146877 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.146200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" event={"ID":"b9f708cc-5ab0-4823-b46f-a34e9c739683","Type":"ContainerStarted","Data":"e4f064186ca84d953893e63efddf80f8745ed428bdbf021a397f3b3d66a68b23"} Jul 13 03:42:42.146877 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.146256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" event={"ID":"b9f708cc-5ab0-4823-b46f-a34e9c739683","Type":"ContainerStarted","Data":"390fa51afa864f84066b0189b2910141a3bd1a16a5f5e12d67c3de2deb945172"} Jul 13 03:42:42.146877 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.146283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" event={"ID":"b9f708cc-5ab0-4823-b46f-a34e9c739683","Type":"ContainerStarted","Data":"bdf523bb776633337c581e0ab52b0d19e9df47b60bca21b4dd628c246466100a"} Jul 13 03:42:42.149682 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.149598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ggwx" event={"ID":"446ecfd0-eaac-47cd-b3a8-6046826dcbac","Type":"ContainerStarted","Data":"bbd621d77ffe0e81c263c8e4bc4b42dfafca5f0eefe134a92eb0b6dfde41aec6"} Jul 13 03:42:42.151617 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.151546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-45z9p" event={"ID":"dbe98aea-97ba-462c-b92d-4d147e56fd4d","Type":"ContainerStarted","Data":"46f083b6e1a67da5b0500a86e52449b4dbe5d9bcd9cf2e1032354b1118d40ce2"} Jul 13 03:42:42.157734 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.154471 2562 generic.go:356] "Generic (PLEG): container finished" podID="443e8e18-faf1-409b-9461-49bc5e1445f5" containerID="bfb29bdcc4d47a11400ba5c1faebcaecae18dbb1958ea158018317c343ea15ae" exitCode=0 Jul 13 03:42:42.157734 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.154612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5t" event={"ID":"443e8e18-faf1-409b-9461-49bc5e1445f5","Type":"ContainerDied","Data":"bfb29bdcc4d47a11400ba5c1faebcaecae18dbb1958ea158018317c343ea15ae"} Jul 13 03:42:42.160369 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.160311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pcmzm" event={"ID":"06cbdc46-0047-4f05-a53a-a9285a2033bd","Type":"ContainerStarted","Data":"4927ae9aaeae2927dde48810453347900559c7890e10ac45777f23e80adfc029"} Jul 13 03:42:42.162957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.162927 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"c6c8b399a93d14a98da58be7c00bbd905fad3788eaeb2f4f3fb0d7625a06d312"} Jul 13 03:42:42.165577 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.164842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"842c1dc849c275748580c4cfde97b2a05866a44e3fad8dd0159dea8337e9392a"} Jul 13 03:42:42.178004 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:42.176544 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q2jt5" podStartSLOduration=2.853688094 podStartE2EDuration="4.176518721s" totalImagesPullingTime="1.322830627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:42.157430871 +0000 UTC m=+66.933388573" watchObservedRunningTime="2026-07-13 03:42:42.176518721 +0000 UTC m=+66.952476421" Jul 13 03:42:43.060168 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.059597 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-xr84f" podStartSLOduration=3.345479117 podStartE2EDuration="5.059575384s" totalImagesPullingTime="1.714096267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:42.192723834 +0000 UTC m=+66.968681537" watchObservedRunningTime="2026-07-13 03:42:43.059575384 +0000 UTC m=+67.835533086" Jul 13 03:42:43.063312 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.061514 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh"] Jul 13 03:42:43.065141 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.065122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.067250 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.066824 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 13 03:42:43.072216 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.069321 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2zxmx\"" Jul 13 03:42:43.072216 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.069527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 13 03:42:43.072216 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.070320 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 13 03:42:43.072216 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.070525 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-mps21d9a6q03\"" Jul 13 03:42:43.072216 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.072159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 03:42:43.079044 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.078959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh"] Jul 13 03:42:43.119631 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-audit-log\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119760 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119760 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-client-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119760 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-tls\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119760 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-metrics-server-audit-profiles\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119946 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-client-certs\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.119946 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.119826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkzj\" (UniqueName: \"kubernetes.io/projected/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-kube-api-access-zrkzj\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.174704 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.174601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5t" event={"ID":"443e8e18-faf1-409b-9461-49bc5e1445f5","Type":"ContainerStarted","Data":"59b99f87858fde33cc811487ac967d6803d0ef945166772a29b42b9d937bb2f4"} Jul 13 03:42:43.174704 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.174665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjb5t" event={"ID":"443e8e18-faf1-409b-9461-49bc5e1445f5","Type":"ContainerStarted","Data":"f4cb649f6cc6178bc5250dd100cd3e853ba4fb1a36ac713d6eaf7f7cdf365b27"} Jul 13 03:42:43.177831 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.177785 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb" exitCode=0 Jul 13 03:42:43.178157 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.178080 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb"} Jul 13 03:42:43.181402 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.180717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ggwx" event={"ID":"446ecfd0-eaac-47cd-b3a8-6046826dcbac","Type":"ContainerStarted","Data":"9a345ef4b9433f4f9229fda79db7ca6569de3fcd4c3be187b42edd148cbd972d"} Jul 13 03:42:43.219924 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.219870 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mjb5t" podStartSLOduration=3.569442554 podStartE2EDuration="5.219847782s" totalImagesPullingTime="1.528436076s" totalInitContainerRuntime="121.969152ms" isStatefulPod=true podCreationTimestamp="2026-07-13 03:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:43.195233748 +0000 UTC m=+67.971191448" watchObservedRunningTime="2026-07-13 03:42:43.219847782 +0000 UTC m=+67.995805486" Jul 13 03:42:43.220538 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.220622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-client-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.220622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-tls\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.220622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-metrics-server-audit-profiles\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.220891 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-client-certs\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.220891 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.220841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkzj\" (UniqueName: \"kubernetes.io/projected/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-kube-api-access-zrkzj\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.221079 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.221061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-audit-log\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.224482 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.224070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-metrics-server-audit-profiles\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.225631 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.225150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-tls\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.225631 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.225420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-audit-log\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.226468 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.226184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.230256 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.230230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-secret-metrics-server-client-certs\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.235071 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.235046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkzj\" (UniqueName: \"kubernetes.io/projected/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-kube-api-access-zrkzj\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.236198 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.236127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4-client-ca-bundle\") pod \"metrics-server-7c7d55bcc7-hqzsh\" (UID: \"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4\") " pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.409185 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.409151 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz"] Jul 13 03:42:43.412669 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.412647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.414669 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.414643 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lp5v4\"" Jul 13 03:42:43.414784 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.414684 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 13 03:42:43.419197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.419155 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:42:43.419840 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.419817 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz"] Jul 13 03:42:43.532890 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.532851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.533070 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.532907 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2r6h\" (UniqueName: \"kubernetes.io/projected/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-kube-api-access-w2r6h\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.634377 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.634337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.634547 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.634393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2r6h\" (UniqueName: \"kubernetes.io/projected/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-kube-api-access-w2r6h\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.635026 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:43.634723 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 13 03:42:43.635026 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:42:43.634799 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert podName:c57e8fba-17c6-47fb-bc01-3d01d2c64e32 nodeName:}" failed. No retries permitted until 2026-07-13 03:42:44.134776044 +0000 UTC m=+68.910733747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-dk4zz" (UID: "c57e8fba-17c6-47fb-bc01-3d01d2c64e32") : secret "monitoring-plugin-cert" not found Jul 13 03:42:43.643000 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.642935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2r6h\" (UniqueName: \"kubernetes.io/projected/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-kube-api-access-w2r6h\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:43.754584 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.754553 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b"] Jul 13 03:42:43.758033 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.758009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.760034 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.760058 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.760451 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.760661 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.760874 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4wcck\"" Jul 13 03:42:43.761293 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.761074 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 13 03:42:43.768652 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.767287 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 13 03:42:43.768652 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.768577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b"] Jul 13 03:42:43.836457 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836457 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6pfr\" (UniqueName: \"kubernetes.io/projected/e531e19e-0200-48ff-b34d-e864cb3e01ea-kube-api-access-n6pfr\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836457 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-federate-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-metrics-client-ca\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-serving-certs-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.836706 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.836658 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937645 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937645 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6pfr\" (UniqueName: \"kubernetes.io/projected/e531e19e-0200-48ff-b34d-e864cb3e01ea-kube-api-access-n6pfr\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-federate-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-metrics-client-ca\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-serving-certs-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.937861 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.937856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.939017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.938652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-serving-certs-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.939017 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.938847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.939450 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.939363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e531e19e-0200-48ff-b34d-e864cb3e01ea-metrics-client-ca\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.940882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.940853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-federate-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.941965 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.941942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-telemeter-client-tls\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.942301 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.942262 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.942469 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.942425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e531e19e-0200-48ff-b34d-e864cb3e01ea-secret-telemeter-client\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:43.946842 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:43.946818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6pfr\" (UniqueName: \"kubernetes.io/projected/e531e19e-0200-48ff-b34d-e864cb3e01ea-kube-api-access-n6pfr\") pod \"telemeter-client-7b7f5f8897-xmr8b\" (UID: \"e531e19e-0200-48ff-b34d-e864cb3e01ea\") " pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:44.072767 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.072571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" Jul 13 03:42:44.095938 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.095050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh"] Jul 13 03:42:44.141005 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.140466 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:44.143846 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.143785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c57e8fba-17c6-47fb-bc01-3d01d2c64e32-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dk4zz\" (UID: \"c57e8fba-17c6-47fb-bc01-3d01d2c64e32\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:44.194380 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.193524 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ggwx" event={"ID":"446ecfd0-eaac-47cd-b3a8-6046826dcbac","Type":"ContainerStarted","Data":"07bde0a2399591ec929fc1221d7e628b65a19e77588c06f85cdf84b803e633f3"} Jul 13 03:42:44.199859 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.199733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"62a21df7d694a6c5302905aeee03150ac2da814ba595722336ef952f7d4432f8"} Jul 13 03:42:44.199859 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.199812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"6433c7376c246ed139c8cc504c2b4b2a640aa1955660240b53f023c0ae782a42"} Jul 13 03:42:44.203078 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.203014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" event={"ID":"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4","Type":"ContainerStarted","Data":"6b29074da71196e33192a107403c94400dd3bfaf53b00f61c10a3d0665cadf85"} Jul 13 03:42:44.211626 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.211221 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8ggwx" podStartSLOduration=66.837550223 podStartE2EDuration="1m8.211202431s" totalImagesPullingTime="1.373652208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:44.209382269 +0000 UTC m=+68.985339969" watchObservedRunningTime="2026-07-13 03:42:44.211202431 +0000 UTC m=+68.987160133" Jul 13 03:42:44.254277 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.254196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b"] Jul 13 03:42:44.328184 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.328015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:44.884287 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.884245 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:42:44.889273 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.889247 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.892528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.892498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.895581 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.895603 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.896104 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.896143 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.896756 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.897141 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 03:42:44.897475 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.897452 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f4tb77ms3dmm7\"" Jul 13 03:42:44.897899 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.897490 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-stzdp\"" Jul 13 03:42:44.897899 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.897821 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 03:42:44.902269 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.898497 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 03:42:44.902269 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.898916 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 03:42:44.902269 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.899657 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 03:42:44.906784 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.902861 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 03:42:44.911518 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.908901 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 03:42:44.911518 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.910613 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:42:44.914022 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.914001 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 03:42:44.948529 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948678 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948678 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948678 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948678 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948847 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948847 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948847 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948939 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948901 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.948995 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.948937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949070 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949192 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949192 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949192 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkkc\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:44.949366 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:44.949341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.050081 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.050081 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.050081 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkkc\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.050950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.051041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.051077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.051158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.051757 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.051454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.052099 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.052054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.052683 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.052655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.054835 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.054807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.056713 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.056372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.057510 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.057075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.057510 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.057302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.057510 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.057460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.057791 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.057770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.059681 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.059653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.059928 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.059907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.060039 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.059999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.060357 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.060316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.060447 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.060368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.060923 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.060904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.061801 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.061752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.062053 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.062009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.063298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.063260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.063649 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.063472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkkc\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc\") pod \"prometheus-k8s-0\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:45.209581 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.209490 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"248c8daa3abab7db74078f42c181381ec9f4dacc10982309dc9864cc5c82bb2a"} Jul 13 03:42:45.228319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:45.228289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:42:46.086300 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:46.086265 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-54vc8" Jul 13 03:42:46.162113 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:46.162082 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:42:47.013213 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:47.013183 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode531e19e_0200_48ff_b34d_e864cb3e01ea.slice/crio-b8f36be3a8d4d8511767de236ff8e7a2bf5305b54b7667abda1357ca7d2d9e05 WatchSource:0}: Error finding container b8f36be3a8d4d8511767de236ff8e7a2bf5305b54b7667abda1357ca7d2d9e05: Status 404 returned error can't find the container with id b8f36be3a8d4d8511767de236ff8e7a2bf5305b54b7667abda1357ca7d2d9e05 Jul 13 03:42:47.221111 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:47.221028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" event={"ID":"e531e19e-0200-48ff-b34d-e864cb3e01ea","Type":"ContainerStarted","Data":"b8f36be3a8d4d8511767de236ff8e7a2bf5305b54b7667abda1357ca7d2d9e05"} Jul 13 03:42:47.722841 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:47.720830 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz"] Jul 13 03:42:47.726514 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:42:47.726461 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57e8fba_17c6_47fb_bc01_3d01d2c64e32.slice/crio-8b97300849b86ba312c10eee9158cb5174942e5ccb25a72e27307dd14dfc8efd WatchSource:0}: Error finding container 8b97300849b86ba312c10eee9158cb5174942e5ccb25a72e27307dd14dfc8efd: Status 404 returned error can't find the container with id 8b97300849b86ba312c10eee9158cb5174942e5ccb25a72e27307dd14dfc8efd Jul 13 03:42:47.759513 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:47.759132 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:42:48.227565 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.227522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" event={"ID":"c57e8fba-17c6-47fb-bc01-3d01d2c64e32","Type":"ContainerStarted","Data":"8b97300849b86ba312c10eee9158cb5174942e5ccb25a72e27307dd14dfc8efd"} Jul 13 03:42:48.229299 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.229262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-45z9p" event={"ID":"dbe98aea-97ba-462c-b92d-4d147e56fd4d","Type":"ContainerStarted","Data":"899eb9ba9763bd8b4983f6a2c2e2a5a292358c8f64358463cfc61d339fbfa46e"} Jul 13 03:42:48.231107 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.231081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pcmzm" event={"ID":"06cbdc46-0047-4f05-a53a-a9285a2033bd","Type":"ContainerStarted","Data":"2439892a7a40dc94ec2b30d02b91e95ba2da3373d69582f25be08a171c211a24"} Jul 13 03:42:48.231235 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.231136 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:42:48.234676 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.234650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"69e121774f0f75399dde1900ae13cdc04131a68200e97f94e9803b9d58dfc1f7"} Jul 13 03:42:48.234791 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.234712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"efca99a3f5bd454b7a29085f9c04875d3588904dc795d4453e11b3e3dda08641"} Jul 13 03:42:48.234791 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.234754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" event={"ID":"babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e","Type":"ContainerStarted","Data":"e23b57e57f780c39c192ffa065abaa0dda5ef410c134ccc426e34eeb64584978"} Jul 13 03:42:48.234905 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.234810 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:48.238122 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.238096 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc"} Jul 13 03:42:48.238222 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.238164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc"} Jul 13 03:42:48.238222 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.238208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca"} Jul 13 03:42:48.238302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.238246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb"} Jul 13 03:42:48.238302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.238270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3"} Jul 13 03:42:48.239686 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.239651 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403" exitCode=0 Jul 13 03:42:48.239779 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.239697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403"} Jul 13 03:42:48.239779 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.239752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"fd43f181303c713a96570f5501c34838c0c433bd5f6a0cf50c1716fff50d2cd7"} Jul 13 03:42:48.243396 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.243343 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-45z9p" podStartSLOduration=66.017922386 podStartE2EDuration="1m12.243314087s" totalImagesPullingTime="6.225391701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:48.242079756 +0000 UTC m=+73.018037482" watchObservedRunningTime="2026-07-13 03:42:48.243314087 +0000 UTC m=+73.019271792" Jul 13 03:42:48.260535 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.260341 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" podStartSLOduration=2.5408429740000003 podStartE2EDuration="8.260056752s" totalImagesPullingTime="5.719213778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:48.258916169 +0000 UTC m=+73.034873906" watchObservedRunningTime="2026-07-13 03:42:48.260056752 +0000 UTC m=+73.036014455" Jul 13 03:42:48.294620 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:48.294523 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pcmzm" podStartSLOduration=66.069529717 podStartE2EDuration="1m12.294508398s" totalImagesPullingTime="6.224978681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:48.292938919 +0000 UTC m=+73.068896621" watchObservedRunningTime="2026-07-13 03:42:48.294508398 +0000 UTC m=+73.070466097" Jul 13 03:42:49.252529 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:49.252485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" event={"ID":"1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4","Type":"ContainerStarted","Data":"64f02477f0599553f1a281a1e9f56c6d2923b7da184ea9a4c6691592c605d1dd"} Jul 13 03:42:49.257268 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:49.257240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerStarted","Data":"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5"} Jul 13 03:42:49.264961 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:49.264934 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b7bbf66d4-tcqnm" Jul 13 03:42:49.271343 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:49.271297 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" podStartSLOduration=1.86614944 podStartE2EDuration="6.271284652s" totalImagesPullingTime="4.405135212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:49.267383032 +0000 UTC m=+74.043340759" watchObservedRunningTime="2026-07-13 03:42:49.271284652 +0000 UTC m=+74.047242352" Jul 13 03:42:49.291064 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:49.290177 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.287300274 podStartE2EDuration="10.290158578s" totalImagesPullingTime="5.981627584s" totalInitContainerRuntime="21.23072ms" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:49.289660289 +0000 UTC m=+74.065618013" watchObservedRunningTime="2026-07-13 03:42:49.290158578 +0000 UTC m=+74.066116279" Jul 13 03:42:50.078169 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.077822 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:50.078735 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.078552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:42:50.080066 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.080000 2562 patch_prober.go:28] interesting pod/console-644df44c57-szpfs container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body= Jul 13 03:42:50.080066 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.080050 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-644df44c57-szpfs" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" Jul 13 03:42:50.264988 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.264930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" event={"ID":"c57e8fba-17c6-47fb-bc01-3d01d2c64e32","Type":"ContainerStarted","Data":"359dd1ecc2499a038830ce213c0ebc58e9e45400f40d2707de0cf769bb6d7601"} Jul 13 03:42:50.265449 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.265102 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:50.267593 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.267534 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" event={"ID":"e531e19e-0200-48ff-b34d-e864cb3e01ea","Type":"ContainerStarted","Data":"9c75e4f9a5f24e1421e418ea4e55b4af01abbe92d9a80bd24d82f4bbbade7601"} Jul 13 03:42:50.267593 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.267592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" event={"ID":"e531e19e-0200-48ff-b34d-e864cb3e01ea","Type":"ContainerStarted","Data":"c85227a35c78d313954b259d7b68cd1a637fbe129b3bb2b73c3bfba0a7bb8fa1"} Jul 13 03:42:50.267769 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.267627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" event={"ID":"e531e19e-0200-48ff-b34d-e864cb3e01ea","Type":"ContainerStarted","Data":"d67d4db81723934212422bf85a988abf79250ceeaa7ece4418092db21abfa440"} Jul 13 03:42:50.270941 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.270923 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" Jul 13 03:42:50.280562 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.280520 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dk4zz" podStartSLOduration=5.109158434 podStartE2EDuration="7.280504183s" totalImagesPullingTime="2.171345749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:50.278776059 +0000 UTC m=+75.054733783" watchObservedRunningTime="2026-07-13 03:42:50.280504183 +0000 UTC m=+75.056461884" Jul 13 03:42:50.316939 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:50.316835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b7f5f8897-xmr8b" podStartSLOduration=4.430776262 podStartE2EDuration="7.316815766s" totalImagesPullingTime="2.886039504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:50.316277968 +0000 UTC m=+75.092235691" watchObservedRunningTime="2026-07-13 03:42:50.316815766 +0000 UTC m=+75.092773467" Jul 13 03:42:51.122203 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:51.122170 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:42:52.280161 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59"} Jul 13 03:42:52.280527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a"} Jul 13 03:42:52.280527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1"} Jul 13 03:42:52.280527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55"} Jul 13 03:42:52.280527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8"} Jul 13 03:42:52.280527 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.280322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerStarted","Data":"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e"} Jul 13 03:42:52.305895 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:52.305839 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.841475022 podStartE2EDuration="8.305817468s" totalImagesPullingTime="3.437385361s" totalInitContainerRuntime="26.957085ms" isStatefulPod=false podCreationTimestamp="2026-07-13 03:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:42:52.303424222 +0000 UTC m=+77.079381923" watchObservedRunningTime="2026-07-13 03:42:52.305817468 +0000 UTC m=+77.081775179" Jul 13 03:42:55.229076 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:42:55.229036 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:43:03.420467 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:03.420358 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:43:03.420467 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:03.420417 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:43:11.185487 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.185436 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b56f77d5f-xj6sr" podUID="63ae9d45-0037-4870-9daa-326801a8afea" containerName="console" containerID="cri-o://452b1ff52da31b8bea1a0b88f046b9e4913152c41b40e3e04fe1e39acf08162c" gracePeriod=15 Jul 13 03:43:11.364770 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.364726 2562 generic.go:356] "Generic (PLEG): container finished" podID="63ae9d45-0037-4870-9daa-326801a8afea" containerID="452b1ff52da31b8bea1a0b88f046b9e4913152c41b40e3e04fe1e39acf08162c" exitCode=0 Jul 13 03:43:11.364926 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.364794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56f77d5f-xj6sr" event={"ID":"63ae9d45-0037-4870-9daa-326801a8afea","Type":"ContainerDied","Data":"452b1ff52da31b8bea1a0b88f046b9e4913152c41b40e3e04fe1e39acf08162c"} Jul 13 03:43:11.430333 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.430313 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:43:11.483326 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483263 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483326 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc76k\" (UniqueName: \"kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483466 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483329 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483466 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483353 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483466 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483387 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483466 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca\") pod \"63ae9d45-0037-4870-9daa-326801a8afea\" (UID: \"63ae9d45-0037-4870-9daa-326801a8afea\") " Jul 13 03:43:11.483741 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483711 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config" (OuterVolumeSpecName: "console-config") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:11.483857 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483812 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:11.483923 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.483855 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca" (OuterVolumeSpecName: "service-ca") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:11.485636 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.485606 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:43:11.485745 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.485670 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k" (OuterVolumeSpecName: "kube-api-access-jc76k") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "kube-api-access-jc76k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:43:11.485745 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.485675 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "63ae9d45-0037-4870-9daa-326801a8afea" (UID: "63ae9d45-0037-4870-9daa-326801a8afea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:43:11.584863 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584839 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-serving-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:11.584863 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584859 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc76k\" (UniqueName: \"kubernetes.io/projected/63ae9d45-0037-4870-9daa-326801a8afea-kube-api-access-jc76k\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:11.585023 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584869 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-console-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:11.585023 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584877 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ae9d45-0037-4870-9daa-326801a8afea-console-oauth-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:11.585023 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584885 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-oauth-serving-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:11.585023 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:11.584892 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ae9d45-0037-4870-9daa-326801a8afea-service-ca\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:12.371280 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:12.371249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56f77d5f-xj6sr" event={"ID":"63ae9d45-0037-4870-9daa-326801a8afea","Type":"ContainerDied","Data":"78d996a799a622eb8c827521b9a96a144c92c5ec18f143fae0c60e04d6c90154"} Jul 13 03:43:12.371667 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:12.371310 2562 scope.go:117] "RemoveContainer" containerID="452b1ff52da31b8bea1a0b88f046b9e4913152c41b40e3e04fe1e39acf08162c" Jul 13 03:43:12.371667 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:12.371254 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56f77d5f-xj6sr" Jul 13 03:43:12.384769 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:12.384748 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:43:12.388845 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:12.388825 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b56f77d5f-xj6sr"] Jul 13 03:43:13.752097 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:13.752065 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ae9d45-0037-4870-9daa-326801a8afea" path="/var/lib/kubelet/pods/63ae9d45-0037-4870-9daa-326801a8afea/volumes" Jul 13 03:43:16.298037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.297951 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-644df44c57-szpfs" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerName="console" containerID="cri-o://b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128" gracePeriod=15 Jul 13 03:43:16.611956 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.611933 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:43:16.624661 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624636 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624738 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624685 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624738 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624720 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624821 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624821 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624779 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624821 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624812 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.624956 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.624911 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx6vx\" (UniqueName: \"kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx\") pod \"2611ecbe-2ee9-4fe5-92ee-06885230266a\" (UID: \"2611ecbe-2ee9-4fe5-92ee-06885230266a\") " Jul 13 03:43:16.625044 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.625022 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca" (OuterVolumeSpecName: "service-ca") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:16.625228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.625206 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-service-ca\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.625313 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.625223 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config" (OuterVolumeSpecName: "console-config") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:16.625313 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.625249 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:16.625443 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.625414 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:43:16.626918 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.626887 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:43:16.627093 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.626949 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:43:16.627093 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.627042 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx" (OuterVolumeSpecName: "kube-api-access-dx6vx") pod "2611ecbe-2ee9-4fe5-92ee-06885230266a" (UID: "2611ecbe-2ee9-4fe5-92ee-06885230266a"). InnerVolumeSpecName "kube-api-access-dx6vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:43:16.725762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725721 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-oauth-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.725762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725757 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-serving-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.725762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725767 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-trusted-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.725957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725776 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-console-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.725957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725786 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2611ecbe-2ee9-4fe5-92ee-06885230266a-oauth-serving-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:16.725957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:16.725794 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx6vx\" (UniqueName: \"kubernetes.io/projected/2611ecbe-2ee9-4fe5-92ee-06885230266a-kube-api-access-dx6vx\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:43:17.395214 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.395178 2562 generic.go:356] "Generic (PLEG): container finished" podID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerID="b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128" exitCode=0 Jul 13 03:43:17.395214 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.395218 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644df44c57-szpfs" Jul 13 03:43:17.395781 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.395262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644df44c57-szpfs" event={"ID":"2611ecbe-2ee9-4fe5-92ee-06885230266a","Type":"ContainerDied","Data":"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128"} Jul 13 03:43:17.395781 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.395310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644df44c57-szpfs" event={"ID":"2611ecbe-2ee9-4fe5-92ee-06885230266a","Type":"ContainerDied","Data":"5ede59c5749b06c87bdec81d3766b94f2431d904255796ce6a2f3807ef773324"} Jul 13 03:43:17.395781 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.395330 2562 scope.go:117] "RemoveContainer" containerID="b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128" Jul 13 03:43:17.406756 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.406731 2562 scope.go:117] "RemoveContainer" containerID="b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128" Jul 13 03:43:17.407047 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:43:17.407025 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128\": container with ID starting with b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128 not found: ID does not exist" containerID="b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128" Jul 13 03:43:17.407127 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.407056 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128"} err="failed to get container status \"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128\": rpc error: code = NotFound desc = could not find container \"b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128\": container with ID starting with b5724308947dd1b6ee60d726ccc321b2828d07711acbf2ac9ebd572e02974128 not found: ID does not exist" Jul 13 03:43:17.416993 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.416951 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:43:17.418625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.418604 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-644df44c57-szpfs"] Jul 13 03:43:17.751496 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:17.751418 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" path="/var/lib/kubelet/pods/2611ecbe-2ee9-4fe5-92ee-06885230266a/volumes" Jul 13 03:43:19.260015 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:19.259967 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pcmzm" Jul 13 03:43:23.425513 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:23.425483 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:43:23.429514 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:23.429486 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c7d55bcc7-hqzsh" Jul 13 03:43:45.228883 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:45.228846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:43:45.248622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:45.248594 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:43:45.523828 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:45.523790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:43:58.935643 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.935609 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.935972 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="alertmanager" containerID="cri-o://003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" gracePeriod=120 Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.936075 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-metric" containerID="cri-o://25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" gracePeriod=120 Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.936163 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy" containerID="cri-o://4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" gracePeriod=120 Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.936106 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-web" containerID="cri-o://86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" gracePeriod=120 Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.936221 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="prom-label-proxy" containerID="cri-o://a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" gracePeriod=120 Jul 13 03:43:58.936404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:58.936151 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="config-reloader" containerID="cri-o://230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" gracePeriod=120 Jul 13 03:43:59.563896 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.563864 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" exitCode=0 Jul 13 03:43:59.563896 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.563887 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" exitCode=0 Jul 13 03:43:59.563896 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.563894 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" exitCode=0 Jul 13 03:43:59.563896 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.563900 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" exitCode=0 Jul 13 03:43:59.564160 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.563930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5"} Jul 13 03:43:59.564160 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.564002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc"} Jul 13 03:43:59.564160 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.564029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb"} Jul 13 03:43:59.564160 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:43:59.564052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3"} Jul 13 03:44:00.185367 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.185343 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.292395 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292368 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292546 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292546 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292449 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292546 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292467 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292546 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292546 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292521 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p67j5\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292785 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292672 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292785 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292708 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292785 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292753 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292953 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292784 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292953 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292797 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 03:44:00.292953 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292953 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.292953 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.292902 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets\") pod \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\" (UID: \"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c\") " Jul 13 03:44:00.293229 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.293192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:00.293287 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.293244 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-main-db\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.294311 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.293719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:00.295711 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.295688 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5" (OuterVolumeSpecName: "kube-api-access-p67j5") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "kube-api-access-p67j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:44:00.296020 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.295952 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume" (OuterVolumeSpecName: "config-volume") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.296362 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.296333 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.296547 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.296496 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out" (OuterVolumeSpecName: "config-out") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 03:44:00.296547 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.296528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.296668 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.296604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.296868 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.296848 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.297175 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.297157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:44:00.300251 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.300215 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.306767 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.306744 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config" (OuterVolumeSpecName: "web-config") pod "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" (UID: "7dd5e715-acf8-4824-bbf4-6a66fcf9c54c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:00.394586 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394563 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394587 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67j5\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-kube-api-access-p67j5\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394602 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-web-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394615 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394629 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394643 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-cluster-tls-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394655 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-metrics-client-ca\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394668 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-main-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394681 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-tls-assets\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394693 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-volume\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394705 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-config-out\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.394970 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.394718 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:00.571153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571074 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" exitCode=0 Jul 13 03:44:00.571153 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571105 2562 generic.go:356] "Generic (PLEG): container finished" podID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerID="86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" exitCode=0 Jul 13 03:44:00.571302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc"} Jul 13 03:44:00.571302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca"} Jul 13 03:44:00.571302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571160 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.571302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dd5e715-acf8-4824-bbf4-6a66fcf9c54c","Type":"ContainerDied","Data":"842c1dc849c275748580c4cfde97b2a05866a44e3fad8dd0159dea8337e9392a"} Jul 13 03:44:00.571302 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.571266 2562 scope.go:117] "RemoveContainer" containerID="a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" Jul 13 03:44:00.579490 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.579472 2562 scope.go:117] "RemoveContainer" containerID="25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" Jul 13 03:44:00.586053 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.586036 2562 scope.go:117] "RemoveContainer" containerID="4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" Jul 13 03:44:00.592955 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.592848 2562 scope.go:117] "RemoveContainer" containerID="86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" Jul 13 03:44:00.593616 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.593591 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:44:00.598241 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.598219 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:44:00.600315 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.600284 2562 scope.go:117] "RemoveContainer" containerID="230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" Jul 13 03:44:00.606862 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.606845 2562 scope.go:117] "RemoveContainer" containerID="003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" Jul 13 03:44:00.613562 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.613547 2562 scope.go:117] "RemoveContainer" containerID="ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb" Jul 13 03:44:00.620500 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.620482 2562 scope.go:117] "RemoveContainer" containerID="a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" Jul 13 03:44:00.620729 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.620712 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5\": container with ID starting with a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5 not found: ID does not exist" containerID="a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" Jul 13 03:44:00.620770 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.620736 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5"} err="failed to get container status \"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5\": rpc error: code = NotFound desc = could not find container \"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5\": container with ID starting with a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5 not found: ID does not exist" Jul 13 03:44:00.620770 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.620762 2562 scope.go:117] "RemoveContainer" containerID="25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" Jul 13 03:44:00.621032 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.621013 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc\": container with ID starting with 25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc not found: ID does not exist" containerID="25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" Jul 13 03:44:00.621100 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621039 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc"} err="failed to get container status \"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc\": rpc error: code = NotFound desc = could not find container \"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc\": container with ID starting with 25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc not found: ID does not exist" Jul 13 03:44:00.621100 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621054 2562 scope.go:117] "RemoveContainer" containerID="4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" Jul 13 03:44:00.621277 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.621260 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc\": container with ID starting with 4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc not found: ID does not exist" containerID="4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" Jul 13 03:44:00.621316 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621283 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc"} err="failed to get container status \"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc\": rpc error: code = NotFound desc = could not find container \"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc\": container with ID starting with 4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc not found: ID does not exist" Jul 13 03:44:00.621316 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621297 2562 scope.go:117] "RemoveContainer" containerID="86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" Jul 13 03:44:00.621524 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.621504 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca\": container with ID starting with 86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca not found: ID does not exist" containerID="86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" Jul 13 03:44:00.621562 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621529 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca"} err="failed to get container status \"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca\": rpc error: code = NotFound desc = could not find container \"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca\": container with ID starting with 86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca not found: ID does not exist" Jul 13 03:44:00.621562 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621543 2562 scope.go:117] "RemoveContainer" containerID="230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" Jul 13 03:44:00.621760 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.621741 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb\": container with ID starting with 230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb not found: ID does not exist" containerID="230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" Jul 13 03:44:00.621831 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621767 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb"} err="failed to get container status \"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb\": rpc error: code = NotFound desc = could not find container \"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb\": container with ID starting with 230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb not found: ID does not exist" Jul 13 03:44:00.621831 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.621799 2562 scope.go:117] "RemoveContainer" containerID="003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" Jul 13 03:44:00.622070 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.622054 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3\": container with ID starting with 003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3 not found: ID does not exist" containerID="003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" Jul 13 03:44:00.622146 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622075 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3"} err="failed to get container status \"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3\": rpc error: code = NotFound desc = could not find container \"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3\": container with ID starting with 003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3 not found: ID does not exist" Jul 13 03:44:00.622146 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622093 2562 scope.go:117] "RemoveContainer" containerID="ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb" Jul 13 03:44:00.622305 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:00.622285 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb\": container with ID starting with ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb not found: ID does not exist" containerID="ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb" Jul 13 03:44:00.622341 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622311 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb"} err="failed to get container status \"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb\": rpc error: code = NotFound desc = could not find container \"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb\": container with ID starting with ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb not found: ID does not exist" Jul 13 03:44:00.622341 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622325 2562 scope.go:117] "RemoveContainer" containerID="a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5" Jul 13 03:44:00.622561 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622542 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5"} err="failed to get container status \"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5\": rpc error: code = NotFound desc = could not find container \"a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5\": container with ID starting with a496ad2138ef611cbc51913b51b321635920977b49351484954114bfa712ccf5 not found: ID does not exist" Jul 13 03:44:00.622605 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622563 2562 scope.go:117] "RemoveContainer" containerID="25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc" Jul 13 03:44:00.622775 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622757 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc"} err="failed to get container status \"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc\": rpc error: code = NotFound desc = could not find container \"25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc\": container with ID starting with 25bf044f31b17f635550fa062fda1d0d85b82c13606c47744b3a68bc1600aedc not found: ID does not exist" Jul 13 03:44:00.622841 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.622776 2562 scope.go:117] "RemoveContainer" containerID="4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc" Jul 13 03:44:00.623073 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623040 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc"} err="failed to get container status \"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc\": rpc error: code = NotFound desc = could not find container \"4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc\": container with ID starting with 4ea2ca68f238cc0eff0629a0beb46197a88886025fd6bdb154221df72da19cdc not found: ID does not exist" Jul 13 03:44:00.623073 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623062 2562 scope.go:117] "RemoveContainer" containerID="86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca" Jul 13 03:44:00.623398 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623368 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca"} err="failed to get container status \"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca\": rpc error: code = NotFound desc = could not find container \"86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca\": container with ID starting with 86cb699a0c0ed2c816037fa95e5f50c9b26eedfba8290071a78594e26cd2beca not found: ID does not exist" Jul 13 03:44:00.623398 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623399 2562 scope.go:117] "RemoveContainer" containerID="230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb" Jul 13 03:44:00.623717 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623690 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb"} err="failed to get container status \"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb\": rpc error: code = NotFound desc = could not find container \"230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb\": container with ID starting with 230ababff715edf9595092982067c40b560837a451a4d7ca0fceb7b78e656acb not found: ID does not exist" Jul 13 03:44:00.623797 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623718 2562 scope.go:117] "RemoveContainer" containerID="003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3" Jul 13 03:44:00.624015 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.623990 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3"} err="failed to get container status \"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3\": rpc error: code = NotFound desc = could not find container \"003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3\": container with ID starting with 003ca19babd0754c0dbd42aedff26e3ef4975222d80a9dc84105a35ae951e4c3 not found: ID does not exist" Jul 13 03:44:00.624118 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.624023 2562 scope.go:117] "RemoveContainer" containerID="ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb" Jul 13 03:44:00.624349 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.624310 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb"} err="failed to get container status \"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb\": rpc error: code = NotFound desc = could not find container \"ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb\": container with ID starting with ca7bd812d655ea2a792551abb8422a0d72ed4840c70595e76457db72eb99e2bb not found: ID does not exist" Jul 13 03:44:00.625243 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625227 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:44:00.625576 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625564 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerName="console" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625579 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerName="console" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625586 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-metric" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625592 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-metric" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625602 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625607 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625615 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="alertmanager" Jul 13 03:44:00.625625 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625620 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="alertmanager" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625628 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="config-reloader" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625634 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="config-reloader" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625642 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="init-config-reloader" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625648 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="init-config-reloader" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625657 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ae9d45-0037-4870-9daa-326801a8afea" containerName="console" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625662 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ae9d45-0037-4870-9daa-326801a8afea" containerName="console" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625667 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-web" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625672 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-web" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625679 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="prom-label-proxy" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625684 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="prom-label-proxy" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625725 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-metric" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625732 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="alertmanager" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625737 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="config-reloader" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625744 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="prom-label-proxy" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625749 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625756 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" containerName="kube-rbac-proxy-web" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625761 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ae9d45-0037-4870-9daa-326801a8afea" containerName="console" Jul 13 03:44:00.625829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.625767 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2611ecbe-2ee9-4fe5-92ee-06885230266a" containerName="console" Jul 13 03:44:00.629120 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.629103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.630989 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.630953 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 13 03:44:00.631093 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.630953 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 13 03:44:00.631093 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631086 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 13 03:44:00.631319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 13 03:44:00.631383 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631374 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kbkqg\"" Jul 13 03:44:00.631651 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631633 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 13 03:44:00.631709 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631644 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 13 03:44:00.631944 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631880 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 13 03:44:00.631944 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.631894 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 13 03:44:00.636683 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.636663 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 13 03:44:00.640517 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.640499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:44:00.798680 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-config-out\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798680 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798785 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrzd\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-kube-api-access-sgrzd\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-web-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.798908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.798995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.799016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.799032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.799115 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.799051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.900137 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.900137 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.900137 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.900137 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-config-out\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.900834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.901304 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.901663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903298 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.903281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrzd\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-kube-api-access-sgrzd\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903813 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.903341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-web-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.903813 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.903421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.904730 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.904703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.905860 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.905100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.905860 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.905297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f7affbe-c560-4b02-b519-15687361374b-config-out\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.905860 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.905591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.905860 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.905793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.906138 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.906036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.906914 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.906886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.907044 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.907010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.907144 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.907122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f7affbe-c560-4b02-b519-15687361374b-web-config\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.909554 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.909533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f7affbe-c560-4b02-b519-15687361374b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.911596 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.911578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrzd\" (UniqueName: \"kubernetes.io/projected/2f7affbe-c560-4b02-b519-15687361374b-kube-api-access-sgrzd\") pod \"alertmanager-main-0\" (UID: \"2f7affbe-c560-4b02-b519-15687361374b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:00.939808 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:00.939790 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 03:44:01.073064 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:01.073033 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 03:44:01.074694 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:44:01.074666 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7affbe_c560_4b02_b519_15687361374b.slice/crio-132d75440915ed67cbe67faf347d8f5210a51e3bd0e5a17928b197a9d9c1a7b2 WatchSource:0}: Error finding container 132d75440915ed67cbe67faf347d8f5210a51e3bd0e5a17928b197a9d9c1a7b2: Status 404 returned error can't find the container with id 132d75440915ed67cbe67faf347d8f5210a51e3bd0e5a17928b197a9d9c1a7b2 Jul 13 03:44:01.577729 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:01.577694 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f7affbe-c560-4b02-b519-15687361374b" containerID="fe77327a88a7514c1975e32b0fbc7669986d8fc25840796de4e7547a615f27f0" exitCode=0 Jul 13 03:44:01.578183 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:01.577783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerDied","Data":"fe77327a88a7514c1975e32b0fbc7669986d8fc25840796de4e7547a615f27f0"} Jul 13 03:44:01.578183 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:01.577854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"132d75440915ed67cbe67faf347d8f5210a51e3bd0e5a17928b197a9d9c1a7b2"} Jul 13 03:44:01.754165 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:01.753527 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd5e715-acf8-4824-bbf4-6a66fcf9c54c" path="/var/lib/kubelet/pods/7dd5e715-acf8-4824-bbf4-6a66fcf9c54c/volumes" Jul 13 03:44:02.585050 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"f6a3ed5738ac85e051c3a5b9c9c9542e2606c8d7f876e279a1c7b63f94c0e11c"} Jul 13 03:44:02.585463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"758b9c851f8b0274b45efae6e6c4ca97f00ed285058dcdb3fe5efb2251b74d48"} Jul 13 03:44:02.585463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"576008ee98fc7cb1113573b688b25b5147d253487e340dd6b7ed68bd7b10b3fc"} Jul 13 03:44:02.585463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"7afc00205df557e11386fcd771ad48390a007d26f33eb1c6572b0e9aed3d569a"} Jul 13 03:44:02.585463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"be6c74cffa72093cc331351cd4914f0e18ccef0d90ae7c50e854ec3de2ca4334"} Jul 13 03:44:02.585463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.585202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f7affbe-c560-4b02-b519-15687361374b","Type":"ContainerStarted","Data":"3536f1e73aa9da7ef0fb6d0ed98ee159a5be7c2c244b6ec01ea115a92f2cc725"} Jul 13 03:44:02.611249 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:02.611194 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5940327720000003 podStartE2EDuration="2.611178159s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.145387ms" isStatefulPod=false podCreationTimestamp="2026-07-13 03:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:44:02.609813813 +0000 UTC m=+147.385771516" watchObservedRunningTime="2026-07-13 03:44:02.611178159 +0000 UTC m=+147.387135859" Jul 13 03:44:03.153377 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.153343 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:03.154037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154001 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="prometheus" containerID="cri-o://a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e" gracePeriod=600 Jul 13 03:44:03.154265 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154038 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="thanos-sidecar" containerID="cri-o://d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55" gracePeriod=600 Jul 13 03:44:03.154385 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154035 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy" containerID="cri-o://9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a" gracePeriod=600 Jul 13 03:44:03.154444 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154067 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-web" containerID="cri-o://0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1" gracePeriod=600 Jul 13 03:44:03.154506 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154066 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-thanos" containerID="cri-o://10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59" gracePeriod=600 Jul 13 03:44:03.154571 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.154092 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="config-reloader" containerID="cri-o://d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8" gracePeriod=600 Jul 13 03:44:03.592246 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592216 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59" exitCode=0 Jul 13 03:44:03.592246 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592242 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a" exitCode=0 Jul 13 03:44:03.592246 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592251 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55" exitCode=0 Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592257 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8" exitCode=0 Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592265 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e" exitCode=0 Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59"} Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a"} Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55"} Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8"} Jul 13 03:44:03.592641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:03.592471 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e"} Jul 13 03:44:04.498463 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.498439 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.536500 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536395 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536633 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536633 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536633 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536619 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536786 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536786 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.536786 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536765 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536791 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536859 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536883 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536918 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.536950 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537000 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537084 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537111 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkkc\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537144 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.537206 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle\") pod \"ed35aecf-04f2-4120-9341-4fde05993950\" (UID: \"ed35aecf-04f2-4120-9341-4fde05993950\") " Jul 13 03:44:04.540058 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.539654 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.541453 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.541423 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config" (OuterVolumeSpecName: "config") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.541932 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.541899 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.542354 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.542067 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 03:44:04.542354 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.542301 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.542691 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.542566 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.542750 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.542712 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.543311 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.543159 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.543529 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.543498 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.543633 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.543554 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.544269 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.544239 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.544369 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.544290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out" (OuterVolumeSpecName: "config-out") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 03:44:04.545865 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.545837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:44:04.545865 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.545844 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.546412 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.546386 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.546656 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.546633 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc" (OuterVolumeSpecName: "kube-api-access-gmkkc") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "kube-api-access-gmkkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:44:04.546656 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.546644 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.546820 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.546784 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 03:44:04.547709 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.547687 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.553549 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.553528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config" (OuterVolumeSpecName: "web-config") pod "ed35aecf-04f2-4120-9341-4fde05993950" (UID: "ed35aecf-04f2-4120-9341-4fde05993950"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 03:44:04.599793 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.599764 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed35aecf-04f2-4120-9341-4fde05993950" containerID="0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1" exitCode=0 Jul 13 03:44:04.600122 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.599836 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.600122 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.599845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1"} Jul 13 03:44:04.600122 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.599942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed35aecf-04f2-4120-9341-4fde05993950","Type":"ContainerDied","Data":"fd43f181303c713a96570f5501c34838c0c433bd5f6a0cf50c1716fff50d2cd7"} Jul 13 03:44:04.600122 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.599998 2562 scope.go:117] "RemoveContainer" containerID="10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59" Jul 13 03:44:04.607378 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.607364 2562 scope.go:117] "RemoveContainer" containerID="9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a" Jul 13 03:44:04.614295 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.614278 2562 scope.go:117] "RemoveContainer" containerID="0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1" Jul 13 03:44:04.621232 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.621213 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:04.621433 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.621416 2562 scope.go:117] "RemoveContainer" containerID="d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55" Jul 13 03:44:04.624814 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.624795 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:04.628165 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.628150 2562 scope.go:117] "RemoveContainer" containerID="d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8" Jul 13 03:44:04.634643 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.634628 2562 scope.go:117] "RemoveContainer" containerID="a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e" Jul 13 03:44:04.638814 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638795 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638818 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638834 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-db\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638849 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638862 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-metrics-client-ca\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638878 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638894 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-kube-rbac-proxy\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.638907 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638905 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed35aecf-04f2-4120-9341-4fde05993950-config-out\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638918 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-tls-assets\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638929 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638944 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638958 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.638971 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-grpc-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639001 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639015 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-metrics-client-certs\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639028 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkkc\" (UniqueName: \"kubernetes.io/projected/ed35aecf-04f2-4120-9341-4fde05993950-kube-api-access-gmkkc\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639042 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639055 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed35aecf-04f2-4120-9341-4fde05993950-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639069 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.639242 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.639082 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed35aecf-04f2-4120-9341-4fde05993950-web-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:44:04.641646 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.641630 2562 scope.go:117] "RemoveContainer" containerID="342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403" Jul 13 03:44:04.648599 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648576 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648860 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-thanos" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648869 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-thanos" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648882 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648888 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648895 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="config-reloader" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648901 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="config-reloader" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648908 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-web" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648914 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-web" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648923 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="thanos-sidecar" Jul 13 03:44:04.648929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648931 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="thanos-sidecar" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648949 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="init-config-reloader" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648953 2562 scope.go:117] "RemoveContainer" containerID="10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.648955 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="init-config-reloader" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649040 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="prometheus" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649049 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="prometheus" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649124 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-web" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649134 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649147 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="prometheus" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649153 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="kube-rbac-proxy-thanos" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649160 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="thanos-sidecar" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649169 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed35aecf-04f2-4120-9341-4fde05993950" containerName="config-reloader" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.649272 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59\": container with ID starting with 10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59 not found: ID does not exist" containerID="10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649293 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59"} err="failed to get container status \"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59\": rpc error: code = NotFound desc = could not find container \"10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59\": container with ID starting with 10b8463424c2a25ca29a6e97694a2f4a52626dc24bda8e8cd35f3021f155bb59 not found: ID does not exist" Jul 13 03:44:04.649320 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649309 2562 scope.go:117] "RemoveContainer" containerID="9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a" Jul 13 03:44:04.649829 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.649545 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a\": container with ID starting with 9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a not found: ID does not exist" containerID="9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a" Jul 13 03:44:04.649829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649570 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a"} err="failed to get container status \"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a\": rpc error: code = NotFound desc = could not find container \"9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a\": container with ID starting with 9d76a7111f761c1980725848787404d002368f8158205c48cbef1616bd56402a not found: ID does not exist" Jul 13 03:44:04.649829 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649586 2562 scope.go:117] "RemoveContainer" containerID="0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1" Jul 13 03:44:04.649934 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.649825 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1\": container with ID starting with 0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1 not found: ID does not exist" containerID="0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1" Jul 13 03:44:04.649934 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649844 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1"} err="failed to get container status \"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1\": rpc error: code = NotFound desc = could not find container \"0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1\": container with ID starting with 0718faa4eafb5870aba958e5d0610a3915595fd91ae78baca66b56e870afd2d1 not found: ID does not exist" Jul 13 03:44:04.649934 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.649857 2562 scope.go:117] "RemoveContainer" containerID="d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55" Jul 13 03:44:04.650149 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.650094 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55\": container with ID starting with d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55 not found: ID does not exist" containerID="d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55" Jul 13 03:44:04.650149 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650112 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55"} err="failed to get container status \"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55\": rpc error: code = NotFound desc = could not find container \"d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55\": container with ID starting with d93c8b58e7fb8a0c1d0b9a5c80c97c89f6ce5867dd4090321c9ea2d526a72b55 not found: ID does not exist" Jul 13 03:44:04.650149 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650124 2562 scope.go:117] "RemoveContainer" containerID="d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8" Jul 13 03:44:04.650372 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.650350 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8\": container with ID starting with d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8 not found: ID does not exist" containerID="d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8" Jul 13 03:44:04.650419 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650379 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8"} err="failed to get container status \"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8\": rpc error: code = NotFound desc = could not find container \"d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8\": container with ID starting with d4e60b83f8d940b5d06d7ccd7efacc8953983f99b0eeec31e6d80eb975b3cdb8 not found: ID does not exist" Jul 13 03:44:04.650419 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650399 2562 scope.go:117] "RemoveContainer" containerID="a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e" Jul 13 03:44:04.650647 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.650631 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e\": container with ID starting with a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e not found: ID does not exist" containerID="a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e" Jul 13 03:44:04.650698 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650650 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e"} err="failed to get container status \"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e\": rpc error: code = NotFound desc = could not find container \"a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e\": container with ID starting with a243d315acbbc573fc79b4e0227d3b1a7af1b656ee44d19d76ae1221f2a1a34e not found: ID does not exist" Jul 13 03:44:04.650698 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650663 2562 scope.go:117] "RemoveContainer" containerID="342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403" Jul 13 03:44:04.650894 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:44:04.650878 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403\": container with ID starting with 342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403 not found: ID does not exist" containerID="342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403" Jul 13 03:44:04.650929 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.650897 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403"} err="failed to get container status \"342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403\": rpc error: code = NotFound desc = could not find container \"342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403\": container with ID starting with 342cdcdd5f11f8aa60b8579366f9278496c0818bf89b6d80b58a8e10a611f403 not found: ID does not exist" Jul 13 03:44:04.655081 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.653287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.655584 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655563 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f4tb77ms3dmm7\"" Jul 13 03:44:04.655733 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 03:44:04.655864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 03:44:04.655864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655839 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 03:44:04.655864 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 03:44:04.656163 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.655877 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 03:44:04.656264 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656248 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 03:44:04.656319 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 03:44:04.656359 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656253 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 03:44:04.656460 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656302 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 03:44:04.656579 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656560 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 03:44:04.656683 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656567 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-stzdp\"" Jul 13 03:44:04.656683 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656668 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 03:44:04.656946 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.656933 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 03:44:04.659480 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.659461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 03:44:04.663092 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.663076 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 03:44:04.666692 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.666671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:04.739545 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739545 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk467\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-kube-api-access-zk467\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.739701 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.739680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740620 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.744045 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.740698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841191 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841191 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841191 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841403 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841560 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841560 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841560 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841560 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841560 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk467\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-kube-api-access-zk467\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.841882 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.842365 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.842365 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.841897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.842365 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.842124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.842517 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.842406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.842517 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.842469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.843092 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.843067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.844614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.844289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.844614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.844289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.844614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.844432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.844614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.844488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.844890 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.844776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.845142 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.845114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.845499 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.845470 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.846003 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.845962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.846528 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.846503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.846671 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.846652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.846712 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.846661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.846854 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.846837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.847245 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.847229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.851774 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.851757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk467\" (UniqueName: \"kubernetes.io/projected/d86a4766-d08a-41c2-a1ec-5da4ac6143b0-kube-api-access-zk467\") pod \"prometheus-k8s-0\" (UID: \"d86a4766-d08a-41c2-a1ec-5da4ac6143b0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:04.963917 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:04.963896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:44:05.094001 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:05.093957 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 03:44:05.095655 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:44:05.095626 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86a4766_d08a_41c2_a1ec_5da4ac6143b0.slice/crio-a57e2581540e40407b59cd1d9a7b669a2dfd7c9b25d066b0d29e8afb0d3284f8 WatchSource:0}: Error finding container a57e2581540e40407b59cd1d9a7b669a2dfd7c9b25d066b0d29e8afb0d3284f8: Status 404 returned error can't find the container with id a57e2581540e40407b59cd1d9a7b669a2dfd7c9b25d066b0d29e8afb0d3284f8 Jul 13 03:44:05.608187 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:05.608152 2562 generic.go:356] "Generic (PLEG): container finished" podID="d86a4766-d08a-41c2-a1ec-5da4ac6143b0" containerID="e6f30e27af718384d8fcbde09a4ce51482bbadc2dd3d103983fe82a085a75f20" exitCode=0 Jul 13 03:44:05.608644 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:05.608227 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerDied","Data":"e6f30e27af718384d8fcbde09a4ce51482bbadc2dd3d103983fe82a085a75f20"} Jul 13 03:44:05.608644 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:05.608305 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"a57e2581540e40407b59cd1d9a7b669a2dfd7c9b25d066b0d29e8afb0d3284f8"} Jul 13 03:44:05.754342 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:05.754316 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed35aecf-04f2-4120-9341-4fde05993950" path="/var/lib/kubelet/pods/ed35aecf-04f2-4120-9341-4fde05993950/volumes" Jul 13 03:44:06.615640 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"bb47ee0e9569730a49d22085ade7c1cc4bccfa44384742696e88ca69636ee25e"} Jul 13 03:44:06.616026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"837d33d95f6ead9c45ddab621b0e500ccff8e01049c75a27c9ebf215236c3904"} Jul 13 03:44:06.616026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615695 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"624ee7f81197afec5e1e797c22ec10ae699eda02ee55d3f613c75965a9eea481"} Jul 13 03:44:06.616026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"06d9e39379ff4f3a7f46975413940a118b428c392ec762502dd784dec7860d75"} Jul 13 03:44:06.616026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"8bf2504fbe5d07a2d5c115c8bfd6334fbe939b66607352e1f91dcfcd67eb9d85"} Jul 13 03:44:06.616026 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.615766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d86a4766-d08a-41c2-a1ec-5da4ac6143b0","Type":"ContainerStarted","Data":"7b7dfca9173a47022f0efc659f8251075d09ea16fabf3ed4c4c36f823f1fd634"} Jul 13 03:44:06.651449 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:06.651377 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6316842510000003 podStartE2EDuration="2.651356569s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.672318ms" isStatefulPod=false podCreationTimestamp="2026-07-13 03:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:44:06.650211431 +0000 UTC m=+151.426169131" watchObservedRunningTime="2026-07-13 03:44:06.651356569 +0000 UTC m=+151.427314271" Jul 13 03:44:09.964607 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:44:09.964576 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:45:04.964382 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:04.964345 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:45:04.979271 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:04.979245 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:45:05.858621 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:05.858593 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 03:45:29.654172 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.654139 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6xfxv"] Jul 13 03:45:29.656425 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.656408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.658148 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.658128 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-47nzs\"" Jul 13 03:45:29.658244 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.658168 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 03:45:29.658590 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.658573 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 03:45:29.665614 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.665590 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6xfxv"] Jul 13 03:45:29.757282 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.757255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jfs\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-kube-api-access-j8jfs\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.757420 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.757299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.858488 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.858461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jfs\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-kube-api-access-j8jfs\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.858589 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.858523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.865761 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.865738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.865908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.865889 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jfs\" (UniqueName: \"kubernetes.io/projected/f5a05472-27cd-4ec7-b381-9d9dbbd5b833-kube-api-access-j8jfs\") pod \"cert-manager-webhook-6486ffd997-6xfxv\" (UID: \"f5a05472-27cd-4ec7-b381-9d9dbbd5b833\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:29.976135 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:29.976076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:30.094290 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:30.094267 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6xfxv"] Jul 13 03:45:30.096857 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:45:30.096826 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a05472_27cd_4ec7_b381_9d9dbbd5b833.slice/crio-f527b6af52681a4f01730fedae8da92d982668d1a1af2cdbfc37f1f5aa30cb51 WatchSource:0}: Error finding container f527b6af52681a4f01730fedae8da92d982668d1a1af2cdbfc37f1f5aa30cb51: Status 404 returned error can't find the container with id f527b6af52681a4f01730fedae8da92d982668d1a1af2cdbfc37f1f5aa30cb51 Jul 13 03:45:30.946581 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:30.946523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" event={"ID":"f5a05472-27cd-4ec7-b381-9d9dbbd5b833","Type":"ContainerStarted","Data":"f527b6af52681a4f01730fedae8da92d982668d1a1af2cdbfc37f1f5aa30cb51"} Jul 13 03:45:33.960346 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:33.960256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" event={"ID":"f5a05472-27cd-4ec7-b381-9d9dbbd5b833","Type":"ContainerStarted","Data":"6eaf3a7217a207efa56b09c0ddaf1b26783de1e4150c8d30357b5bcb91edb44d"} Jul 13 03:45:33.960346 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:33.960301 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:33.974822 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:33.974778 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" podStartSLOduration=2.040729813 podStartE2EDuration="4.974763138s" totalImagesPullingTime="2.934033325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:45:33.973536278 +0000 UTC m=+238.749494002" watchObservedRunningTime="2026-07-13 03:45:33.974763138 +0000 UTC m=+238.750720838" Jul 13 03:45:39.967028 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:39.966971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6xfxv" Jul 13 03:45:41.124309 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.124275 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hsm7l"] Jul 13 03:45:41.127656 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.127639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.129556 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.129536 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tcnz6\"" Jul 13 03:45:41.135323 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.135303 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hsm7l"] Jul 13 03:45:41.150329 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.150301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkcx\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-kube-api-access-crkcx\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.150420 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.150365 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-bound-sa-token\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.251732 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.251708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkcx\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-kube-api-access-crkcx\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.251825 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.251741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-bound-sa-token\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.259127 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.259095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-bound-sa-token\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.259308 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.259289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkcx\" (UniqueName: \"kubernetes.io/projected/52948910-304d-4f95-9357-ce6e43627c89-kube-api-access-crkcx\") pod \"cert-manager-55b4d954d5-hsm7l\" (UID: \"52948910-304d-4f95-9357-ce6e43627c89\") " pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.436678 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.436592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hsm7l" Jul 13 03:45:41.586073 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.586036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hsm7l"] Jul 13 03:45:41.589300 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:45:41.589271 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52948910_304d_4f95_9357_ce6e43627c89.slice/crio-161a000d15a95908e5829e95a5a9a9873ac07d775c0f9924cd6c4be8e6cf599c WatchSource:0}: Error finding container 161a000d15a95908e5829e95a5a9a9873ac07d775c0f9924cd6c4be8e6cf599c: Status 404 returned error can't find the container with id 161a000d15a95908e5829e95a5a9a9873ac07d775c0f9924cd6c4be8e6cf599c Jul 13 03:45:41.992123 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.992089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hsm7l" event={"ID":"52948910-304d-4f95-9357-ce6e43627c89","Type":"ContainerStarted","Data":"b603df4a317aa32b843f1bd992039a01dee2f4eb931de2cac9b10808aff2f4cb"} Jul 13 03:45:41.992123 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:41.992127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hsm7l" event={"ID":"52948910-304d-4f95-9357-ce6e43627c89","Type":"ContainerStarted","Data":"161a000d15a95908e5829e95a5a9a9873ac07d775c0f9924cd6c4be8e6cf599c"} Jul 13 03:45:42.006394 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:45:42.006347 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hsm7l" podStartSLOduration=1.006332447 podStartE2EDuration="1.006332447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:45:42.005024358 +0000 UTC m=+246.780982076" watchObservedRunningTime="2026-07-13 03:45:42.006332447 +0000 UTC m=+246.782290147" Jul 13 03:46:02.166735 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.166662 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj"] Jul 13 03:46:02.170336 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.170319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.173552 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.173527 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 03:46:02.173684 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.173527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 03:46:02.173684 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.173589 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 03:46:02.175135 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.175117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dz94k\"" Jul 13 03:46:02.176535 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.176515 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 03:46:02.189452 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.189399 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj"] Jul 13 03:46:02.317001 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.316946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-tmp\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.317164 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.317043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfttz\" (UniqueName: \"kubernetes.io/projected/9c242bb3-eb92-400a-bbd3-97579b45a795-kube-api-access-sfttz\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.317164 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.317088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.317164 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.317113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-manifests\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.317305 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.317172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418447 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418447 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-tmp\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418697 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfttz\" (UniqueName: \"kubernetes.io/projected/9c242bb3-eb92-400a-bbd3-97579b45a795-kube-api-access-sfttz\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418697 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418697 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-manifests\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.418994 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.418952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-manifests\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.426540 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.426513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c242bb3-eb92-400a-bbd3-97579b45a795-tmp\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.426661 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.426608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.426716 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.426678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c242bb3-eb92-400a-bbd3-97579b45a795-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.428294 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.428279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfttz\" (UniqueName: \"kubernetes.io/projected/9c242bb3-eb92-400a-bbd3-97579b45a795-kube-api-access-sfttz\") pod \"opendatahub-operator-controller-manager-6d6757d97b-xkwsj\" (UID: \"9c242bb3-eb92-400a-bbd3-97579b45a795\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.481940 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.481914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:02.619718 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:02.619673 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj"] Jul 13 03:46:02.622791 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:46:02.622766 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c242bb3_eb92_400a_bbd3_97579b45a795.slice/crio-c3482bd9de59a2f04c67c18ee5350071f09c14dda6d2a3f0e6eb04d35108224e WatchSource:0}: Error finding container c3482bd9de59a2f04c67c18ee5350071f09c14dda6d2a3f0e6eb04d35108224e: Status 404 returned error can't find the container with id c3482bd9de59a2f04c67c18ee5350071f09c14dda6d2a3f0e6eb04d35108224e Jul 13 03:46:03.074952 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:03.074915 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" event={"ID":"9c242bb3-eb92-400a-bbd3-97579b45a795","Type":"ContainerStarted","Data":"c3482bd9de59a2f04c67c18ee5350071f09c14dda6d2a3f0e6eb04d35108224e"} Jul 13 03:46:06.090637 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:06.090600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" event={"ID":"9c242bb3-eb92-400a-bbd3-97579b45a795","Type":"ContainerStarted","Data":"c659b420be9b51549aca191097e04d3dbbbd3e2820470f0fe6127812c040c204"} Jul 13 03:46:07.096145 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:07.096105 2562 generic.go:356] "Generic (PLEG): container finished" podID="9c242bb3-eb92-400a-bbd3-97579b45a795" containerID="c659b420be9b51549aca191097e04d3dbbbd3e2820470f0fe6127812c040c204" exitCode=0 Jul 13 03:46:07.096492 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:07.096193 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" event={"ID":"9c242bb3-eb92-400a-bbd3-97579b45a795","Type":"ContainerDied","Data":"c659b420be9b51549aca191097e04d3dbbbd3e2820470f0fe6127812c040c204"} Jul 13 03:46:08.102575 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:08.102542 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" event={"ID":"9c242bb3-eb92-400a-bbd3-97579b45a795","Type":"ContainerStarted","Data":"19ee5f5651f73d8c8c77a5b572668261f1af3883a435bb7d38b6aa2f3e2aba46"} Jul 13 03:46:08.102996 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:08.102656 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:08.121766 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:08.121710 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" podStartSLOduration=1.697105368 podStartE2EDuration="6.121695704s" totalImagesPullingTime="3.383598762s" totalInitContainerRuntime="1.040991574s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:46:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:46:08.119593314 +0000 UTC m=+272.895551032" watchObservedRunningTime="2026-07-13 03:46:08.121695704 +0000 UTC m=+272.897653403" Jul 13 03:46:19.108747 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:19.108718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6757d97b-xkwsj" Jul 13 03:46:21.384188 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.384156 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4"] Jul 13 03:46:21.392408 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.392373 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.395559 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.395532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 13 03:46:21.395559 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.395547 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 13 03:46:21.395767 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.395565 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 13 03:46:21.395832 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.395810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9vkz4\"" Jul 13 03:46:21.396066 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.396046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 13 03:46:21.396303 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.396275 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 13 03:46:21.397632 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.397612 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4"] Jul 13 03:46:21.579339 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.579306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.579508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.579351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0fb43e-d0f5-4454-b995-47864bcf0927-manager-config\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.579508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.579385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/cf0fb43e-d0f5-4454-b995-47864bcf0927-kube-api-access-zt6cc\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.579508 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.579409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-metrics-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.680639 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.680568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.680639 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.680611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0fb43e-d0f5-4454-b995-47864bcf0927-manager-config\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.680639 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.680634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/cf0fb43e-d0f5-4454-b995-47864bcf0927-kube-api-access-zt6cc\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.680821 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.680752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-metrics-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.681288 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.681268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cf0fb43e-d0f5-4454-b995-47864bcf0927-manager-config\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.683228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.683196 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-metrics-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.683340 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.683249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf0fb43e-d0f5-4454-b995-47864bcf0927-cert\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.692579 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.692540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/cf0fb43e-d0f5-4454-b995-47864bcf0927-kube-api-access-zt6cc\") pod \"lws-controller-manager-6c9556dbfd-9r4f4\" (UID: \"cf0fb43e-d0f5-4454-b995-47864bcf0927\") " pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.702874 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.702853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:21.865997 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:21.865944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4"] Jul 13 03:46:21.869130 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:46:21.869103 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0fb43e_d0f5_4454_b995_47864bcf0927.slice/crio-3d3d0059409960d8b030dcaba475dbb5a0463ac9da9ebc47dd8cd6a70dae16a3 WatchSource:0}: Error finding container 3d3d0059409960d8b030dcaba475dbb5a0463ac9da9ebc47dd8cd6a70dae16a3: Status 404 returned error can't find the container with id 3d3d0059409960d8b030dcaba475dbb5a0463ac9da9ebc47dd8cd6a70dae16a3 Jul 13 03:46:22.165000 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:22.164949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" event={"ID":"cf0fb43e-d0f5-4454-b995-47864bcf0927","Type":"ContainerStarted","Data":"3d3d0059409960d8b030dcaba475dbb5a0463ac9da9ebc47dd8cd6a70dae16a3"} Jul 13 03:46:25.186495 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:25.186450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" event={"ID":"cf0fb43e-d0f5-4454-b995-47864bcf0927","Type":"ContainerStarted","Data":"531883c918d70dc0f41e40afeb3eced5b055168d6b46a4d3fd6fd695d919639c"} Jul 13 03:46:25.186878 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:25.186566 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:46:25.201694 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:25.201652 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" podStartSLOduration=1.819641089 podStartE2EDuration="4.201638801s" totalImagesPullingTime="2.381997712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:46:25.200970832 +0000 UTC m=+289.976928533" watchObservedRunningTime="2026-07-13 03:46:25.201638801 +0000 UTC m=+289.977596559" Jul 13 03:46:26.675430 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.675388 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx"] Jul 13 03:46:26.678908 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.678889 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.680918 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.680896 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zdkr7\"" Jul 13 03:46:26.681082 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.680942 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 03:46:26.681402 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.681385 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 03:46:26.681402 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.681391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 03:46:26.681552 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.681391 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 03:46:26.686514 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.686199 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx"] Jul 13 03:46:26.723995 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.723951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409f770-75df-4e5b-8832-0fba80212182-tls-certs\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.724095 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.724044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2409f770-75df-4e5b-8832-0fba80212182-tmp\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.724170 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.724151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5h8m\" (UniqueName: \"kubernetes.io/projected/2409f770-75df-4e5b-8832-0fba80212182-kube-api-access-s5h8m\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.825309 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.825286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2409f770-75df-4e5b-8832-0fba80212182-tmp\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.825408 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.825337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5h8m\" (UniqueName: \"kubernetes.io/projected/2409f770-75df-4e5b-8832-0fba80212182-kube-api-access-s5h8m\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.825408 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.825371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409f770-75df-4e5b-8832-0fba80212182-tls-certs\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.827600 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.827572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2409f770-75df-4e5b-8832-0fba80212182-tmp\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.827860 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.827843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409f770-75df-4e5b-8832-0fba80212182-tls-certs\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.832356 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.832336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5h8m\" (UniqueName: \"kubernetes.io/projected/2409f770-75df-4e5b-8832-0fba80212182-kube-api-access-s5h8m\") pod \"kube-auth-proxy-7bfd7dd56-9qsvx\" (UID: \"2409f770-75df-4e5b-8832-0fba80212182\") " pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:26.989430 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:26.989367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" Jul 13 03:46:27.112031 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:27.112006 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx"] Jul 13 03:46:27.114867 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:46:27.114843 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2409f770_75df_4e5b_8832_0fba80212182.slice/crio-2dece8adb01265682e5275a523df891876807a96d516ae27a3cc8ce64c966066 WatchSource:0}: Error finding container 2dece8adb01265682e5275a523df891876807a96d516ae27a3cc8ce64c966066: Status 404 returned error can't find the container with id 2dece8adb01265682e5275a523df891876807a96d516ae27a3cc8ce64c966066 Jul 13 03:46:27.195657 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:27.195629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" event={"ID":"2409f770-75df-4e5b-8832-0fba80212182","Type":"ContainerStarted","Data":"2dece8adb01265682e5275a523df891876807a96d516ae27a3cc8ce64c966066"} Jul 13 03:46:30.211073 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:30.211037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" event={"ID":"2409f770-75df-4e5b-8832-0fba80212182","Type":"ContainerStarted","Data":"e43c8ce0d865e948fe7f367e47e19baa85f69b7a0a7c224f25a20400598b6a3e"} Jul 13 03:46:30.228194 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:30.228144 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bfd7dd56-9qsvx" podStartSLOduration=1.698632345 podStartE2EDuration="4.228128069s" totalImagesPullingTime="2.529495724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:46:30.227098087 +0000 UTC m=+295.003055808" watchObservedRunningTime="2026-07-13 03:46:30.228128069 +0000 UTC m=+295.004085768" Jul 13 03:46:35.619684 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:35.619653 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:46:35.620123 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:35.619748 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:46:35.625499 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:35.625482 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 03:46:36.192707 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:46:36.192678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c9556dbfd-9r4f4" Jul 13 03:48:03.574351 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.574317 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5"] Jul 13 03:48:03.577674 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.577655 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:03.579543 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.579513 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 03:48:03.579641 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.579584 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 03:48:03.580052 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.580035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g6nxw\"" Jul 13 03:48:03.580104 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.580049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 03:48:03.587133 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.587111 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5"] Jul 13 03:48:03.730592 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.730565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kddk\" (UniqueName: \"kubernetes.io/projected/00f68645-ffb7-4117-968b-3ab8b4d2886d-kube-api-access-9kddk\") pod \"dns-operator-controller-manager-864dd78c4c-jr9w5\" (UID: \"00f68645-ffb7-4117-968b-3ab8b4d2886d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:03.831206 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.831139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kddk\" (UniqueName: \"kubernetes.io/projected/00f68645-ffb7-4117-968b-3ab8b4d2886d-kube-api-access-9kddk\") pod \"dns-operator-controller-manager-864dd78c4c-jr9w5\" (UID: \"00f68645-ffb7-4117-968b-3ab8b4d2886d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:03.844562 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.844532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kddk\" (UniqueName: \"kubernetes.io/projected/00f68645-ffb7-4117-968b-3ab8b4d2886d-kube-api-access-9kddk\") pod \"dns-operator-controller-manager-864dd78c4c-jr9w5\" (UID: \"00f68645-ffb7-4117-968b-3ab8b4d2886d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:03.887992 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:03.887962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:04.020461 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.020369 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5"] Jul 13 03:48:04.022795 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:48:04.022766 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f68645_ffb7_4117_968b_3ab8b4d2886d.slice/crio-2a6963a179edb1da756111ddb127e17e3d83f7999018bf5e1361947c53128194 WatchSource:0}: Error finding container 2a6963a179edb1da756111ddb127e17e3d83f7999018bf5e1361947c53128194: Status 404 returned error can't find the container with id 2a6963a179edb1da756111ddb127e17e3d83f7999018bf5e1361947c53128194 Jul 13 03:48:04.024791 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.024775 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 03:48:04.628390 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.628348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" event={"ID":"00f68645-ffb7-4117-968b-3ab8b4d2886d","Type":"ContainerStarted","Data":"2a6963a179edb1da756111ddb127e17e3d83f7999018bf5e1361947c53128194"} Jul 13 03:48:04.962341 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.962267 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd"] Jul 13 03:48:04.965592 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.965570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:04.967787 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.967764 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 13 03:48:04.967899 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.967819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s9h8z\"" Jul 13 03:48:04.967899 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.967858 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 13 03:48:04.976887 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:04.976866 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd"] Jul 13 03:48:05.042623 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.042589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec92b4be-4f15-4aa2-8901-3113a1458f6a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.042776 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.042676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.042819 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.042806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kz6\" (UniqueName: \"kubernetes.io/projected/ec92b4be-4f15-4aa2-8901-3113a1458f6a-kube-api-access-j5kz6\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.144273 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.144235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec92b4be-4f15-4aa2-8901-3113a1458f6a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.144495 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.144419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.144620 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.144541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kz6\" (UniqueName: \"kubernetes.io/projected/ec92b4be-4f15-4aa2-8901-3113a1458f6a-kube-api-access-j5kz6\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.144719 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:48:05.144681 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 13 03:48:05.144782 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:48:05.144756 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert podName:ec92b4be-4f15-4aa2-8901-3113a1458f6a nodeName:}" failed. No retries permitted until 2026-07-13 03:48:05.644732554 +0000 UTC m=+390.420690250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-9ghzd" (UID: "ec92b4be-4f15-4aa2-8901-3113a1458f6a") : secret "plugin-serving-cert" not found Jul 13 03:48:05.145039 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.145016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec92b4be-4f15-4aa2-8901-3113a1458f6a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.155210 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.155187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kz6\" (UniqueName: \"kubernetes.io/projected/ec92b4be-4f15-4aa2-8901-3113a1458f6a-kube-api-access-j5kz6\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.648588 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.648556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.650828 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.650807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec92b4be-4f15-4aa2-8901-3113a1458f6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ghzd\" (UID: \"ec92b4be-4f15-4aa2-8901-3113a1458f6a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:05.877214 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:05.877183 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" Jul 13 03:48:06.119739 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:06.119721 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd"] Jul 13 03:48:06.121599 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:48:06.121570 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec92b4be_4f15_4aa2_8901_3113a1458f6a.slice/crio-d9a1bcc4ab356c38673dfcef3f3ba240b6c6dab0ceafb6d03c63684820f35daa WatchSource:0}: Error finding container d9a1bcc4ab356c38673dfcef3f3ba240b6c6dab0ceafb6d03c63684820f35daa: Status 404 returned error can't find the container with id d9a1bcc4ab356c38673dfcef3f3ba240b6c6dab0ceafb6d03c63684820f35daa Jul 13 03:48:06.643080 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:06.643043 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" event={"ID":"00f68645-ffb7-4117-968b-3ab8b4d2886d","Type":"ContainerStarted","Data":"b1705c9b93cc507cb7dd71a28c49502f1f4978efedc63fd2e058173bc98f43fe"} Jul 13 03:48:06.643286 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:06.643140 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:06.644155 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:06.644132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" event={"ID":"ec92b4be-4f15-4aa2-8901-3113a1458f6a","Type":"ContainerStarted","Data":"d9a1bcc4ab356c38673dfcef3f3ba240b6c6dab0ceafb6d03c63684820f35daa"} Jul 13 03:48:06.659904 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:06.659850 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" podStartSLOduration=1.634858471 podStartE2EDuration="3.659835334s" totalImagesPullingTime="2.024976863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:48:06.65765321 +0000 UTC m=+391.433610910" watchObservedRunningTime="2026-07-13 03:48:06.659835334 +0000 UTC m=+391.435793034" Jul 13 03:48:17.651759 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:17.651725 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jr9w5" Jul 13 03:48:28.761905 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:28.761867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" event={"ID":"ec92b4be-4f15-4aa2-8901-3113a1458f6a","Type":"ContainerStarted","Data":"005ac1f9475fe30852cdbf74c3721b39249bfce01f2f34053c3ff9b4384f67fc"} Jul 13 03:48:28.779579 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:28.779534 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ghzd" podStartSLOduration=2.93599166 podStartE2EDuration="24.779519043s" totalImagesPullingTime="21.843527383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:48:28.77847248 +0000 UTC m=+413.554430193" watchObservedRunningTime="2026-07-13 03:48:28.779519043 +0000 UTC m=+413.555476743" Jul 13 03:48:51.821619 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:51.821587 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:48:51.879570 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:51.879539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:48:51.879570 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:51.879577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:48:51.879752 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:51.879650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:51.881550 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:51.881525 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 03:48:52.066579 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.066547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q8j\" (UniqueName: \"kubernetes.io/projected/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-kube-api-access-m8q8j\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.066579 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.066584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-config-file\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.168067 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.167972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q8j\" (UniqueName: \"kubernetes.io/projected/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-kube-api-access-m8q8j\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.168067 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.168034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-config-file\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.168576 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.168556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-config-file\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.175165 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.175141 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q8j\" (UniqueName: \"kubernetes.io/projected/9b88b8bc-fb3a-4792-99c1-0b44dba5b96d-kube-api-access-m8q8j\") pod \"limitador-limitador-54d6d988d5-hfffn\" (UID: \"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.190074 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.190045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:52.502169 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.502110 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:48:52.504932 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:48:52.504908 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b88b8bc_fb3a_4792_99c1_0b44dba5b96d.slice/crio-6ea53315d156a55b7b463f471c884ba0c977eb0df72b71a0c90a4a41789cfbb0 WatchSource:0}: Error finding container 6ea53315d156a55b7b463f471c884ba0c977eb0df72b71a0c90a4a41789cfbb0: Status 404 returned error can't find the container with id 6ea53315d156a55b7b463f471c884ba0c977eb0df72b71a0c90a4a41789cfbb0 Jul 13 03:48:52.869690 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:52.869628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" event={"ID":"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d","Type":"ContainerStarted","Data":"6ea53315d156a55b7b463f471c884ba0c977eb0df72b71a0c90a4a41789cfbb0"} Jul 13 03:48:56.893921 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:56.893883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" event={"ID":"9b88b8bc-fb3a-4792-99c1-0b44dba5b96d","Type":"ContainerStarted","Data":"0580b97213e2f065416b5a81515475a095d9ccd1f268926352c3f430e1368ede"} Jul 13 03:48:56.894350 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:56.894031 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:48:56.914075 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:48:56.913962 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" podStartSLOduration=2.196064758 podStartE2EDuration="5.913947218s" totalImagesPullingTime="3.71788246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:48:56.912094205 +0000 UTC m=+441.688051906" watchObservedRunningTime="2026-07-13 03:48:56.913947218 +0000 UTC m=+441.689904918" Jul 13 03:49:07.900821 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:07.900734 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hfffn" Jul 13 03:49:26.770202 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.770167 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:26.794133 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.794101 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:26.794264 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.794189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:26.796117 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.796096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zq7lr\"" Jul 13 03:49:26.863234 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.863207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxr8g\" (UniqueName: \"kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g\") pod \"authorino-74554c8f88-prb6k\" (UID: \"d044a6f8-26e4-4184-b14b-a1e583c09208\") " pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:26.963762 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.963731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr8g\" (UniqueName: \"kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g\") pod \"authorino-74554c8f88-prb6k\" (UID: \"d044a6f8-26e4-4184-b14b-a1e583c09208\") " pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:26.971132 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:26.971101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr8g\" (UniqueName: \"kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g\") pod \"authorino-74554c8f88-prb6k\" (UID: \"d044a6f8-26e4-4184-b14b-a1e583c09208\") " pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:27.005836 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:27.005809 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:27.006082 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:27.006068 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:27.145430 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:27.145398 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:27.149199 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:49:27.149174 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd044a6f8_26e4_4184_b14b_a1e583c09208.slice/crio-58625343471b681333f4b86549f36ff69ad747040cfbb78e7d700bf9c95830a9 WatchSource:0}: Error finding container 58625343471b681333f4b86549f36ff69ad747040cfbb78e7d700bf9c95830a9: Status 404 returned error can't find the container with id 58625343471b681333f4b86549f36ff69ad747040cfbb78e7d700bf9c95830a9 Jul 13 03:49:28.037465 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:28.037432 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74554c8f88-prb6k" event={"ID":"d044a6f8-26e4-4184-b14b-a1e583c09208","Type":"ContainerStarted","Data":"58625343471b681333f4b86549f36ff69ad747040cfbb78e7d700bf9c95830a9"} Jul 13 03:49:31.054199 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.054159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74554c8f88-prb6k" event={"ID":"d044a6f8-26e4-4184-b14b-a1e583c09208","Type":"ContainerStarted","Data":"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f"} Jul 13 03:49:31.054592 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.054197 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-74554c8f88-prb6k" podUID="d044a6f8-26e4-4184-b14b-a1e583c09208" containerName="authorino" containerID="cri-o://3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f" gracePeriod=30 Jul 13 03:49:31.068832 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.068789 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74554c8f88-prb6k" podStartSLOduration=2.092656031 podStartE2EDuration="5.068769523s" totalImagesPullingTime="2.976113492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:49:31.067064944 +0000 UTC m=+475.843022645" watchObservedRunningTime="2026-07-13 03:49:31.068769523 +0000 UTC m=+475.844727223" Jul 13 03:49:31.299271 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.299248 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:31.409543 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.409478 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxr8g\" (UniqueName: \"kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g\") pod \"d044a6f8-26e4-4184-b14b-a1e583c09208\" (UID: \"d044a6f8-26e4-4184-b14b-a1e583c09208\") " Jul 13 03:49:31.411552 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.411520 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g" (OuterVolumeSpecName: "kube-api-access-mxr8g") pod "d044a6f8-26e4-4184-b14b-a1e583c09208" (UID: "d044a6f8-26e4-4184-b14b-a1e583c09208"). InnerVolumeSpecName "kube-api-access-mxr8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 03:49:31.510710 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:31.510685 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxr8g\" (UniqueName: \"kubernetes.io/projected/d044a6f8-26e4-4184-b14b-a1e583c09208-kube-api-access-mxr8g\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 03:49:32.059492 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.059461 2562 generic.go:356] "Generic (PLEG): container finished" podID="d044a6f8-26e4-4184-b14b-a1e583c09208" containerID="3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f" exitCode=0 Jul 13 03:49:32.059875 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.059502 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74554c8f88-prb6k" Jul 13 03:49:32.059875 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.059518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74554c8f88-prb6k" event={"ID":"d044a6f8-26e4-4184-b14b-a1e583c09208","Type":"ContainerDied","Data":"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f"} Jul 13 03:49:32.059875 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.059548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74554c8f88-prb6k" event={"ID":"d044a6f8-26e4-4184-b14b-a1e583c09208","Type":"ContainerDied","Data":"58625343471b681333f4b86549f36ff69ad747040cfbb78e7d700bf9c95830a9"} Jul 13 03:49:32.059875 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.059564 2562 scope.go:117] "RemoveContainer" containerID="3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f" Jul 13 03:49:32.067870 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.067846 2562 scope.go:117] "RemoveContainer" containerID="3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f" Jul 13 03:49:32.068135 ip-10-0-133-120 kubenswrapper[2562]: E0713 03:49:32.068115 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f\": container with ID starting with 3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f not found: ID does not exist" containerID="3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f" Jul 13 03:49:32.068197 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.068143 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f"} err="failed to get container status \"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f\": rpc error: code = NotFound desc = could not find container \"3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f\": container with ID starting with 3ad01fdb1799024c1a2ae6e1d1f7a8cba9b0ac70b2b5a734f5c795dbd97aba7f not found: ID does not exist" Jul 13 03:49:32.077024 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.077003 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:32.081586 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:32.081567 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74554c8f88-prb6k"] Jul 13 03:49:33.751926 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:49:33.751894 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d044a6f8-26e4-4184-b14b-a1e583c09208" path="/var/lib/kubelet/pods/d044a6f8-26e4-4184-b14b-a1e583c09208/volumes" Jul 13 03:50:07.728109 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:50:07.728073 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:50:55.589051 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:50:55.589021 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:01.606228 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:01.606183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:12.175192 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.175157 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64"] Jul 13 03:51:12.175640 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.175506 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d044a6f8-26e4-4184-b14b-a1e583c09208" containerName="authorino" Jul 13 03:51:12.175640 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.175517 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d044a6f8-26e4-4184-b14b-a1e583c09208" containerName="authorino" Jul 13 03:51:12.175640 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.175578 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d044a6f8-26e4-4184-b14b-a1e583c09208" containerName="authorino" Jul 13 03:51:12.178660 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.178641 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.180697 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.180676 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 03:51:12.180697 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.180689 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 03:51:12.181097 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.181083 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w2wk8\"" Jul 13 03:51:12.181159 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.181144 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 13 03:51:12.186545 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.186525 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64"] Jul 13 03:51:12.215162 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.215272 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.215272 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq544\" (UniqueName: \"kubernetes.io/projected/af143b89-583f-4e84-9299-39ce835815c8-kube-api-access-vq544\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.215272 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af143b89-583f-4e84-9299-39ce835815c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.215272 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.215408 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.215272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316069 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316069 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316278 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq544\" (UniqueName: \"kubernetes.io/projected/af143b89-583f-4e84-9299-39ce835815c8-kube-api-access-vq544\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316278 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af143b89-583f-4e84-9299-39ce835815c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316278 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316278 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316478 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316415 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316643 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.316770 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.316649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.318425 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.318406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af143b89-583f-4e84-9299-39ce835815c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.318591 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.318576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af143b89-583f-4e84-9299-39ce835815c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.323957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.323936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq544\" (UniqueName: \"kubernetes.io/projected/af143b89-583f-4e84-9299-39ce835815c8-kube-api-access-vq544\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx64\" (UID: \"af143b89-583f-4e84-9299-39ce835815c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.489904 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.489819 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:12.615386 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:12.615362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64"] Jul 13 03:51:12.617473 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:51:12.617444 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf143b89_583f_4e84_9299_39ce835815c8.slice/crio-6e8a92986f0a735b87ac07cdea5e1150254fcd239778e6646e46984a7d849449 WatchSource:0}: Error finding container 6e8a92986f0a735b87ac07cdea5e1150254fcd239778e6646e46984a7d849449: Status 404 returned error can't find the container with id 6e8a92986f0a735b87ac07cdea5e1150254fcd239778e6646e46984a7d849449 Jul 13 03:51:13.491967 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:13.491930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:13.522163 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:13.522126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" event={"ID":"af143b89-583f-4e84-9299-39ce835815c8","Type":"ContainerStarted","Data":"6e8a92986f0a735b87ac07cdea5e1150254fcd239778e6646e46984a7d849449"} Jul 13 03:51:14.529194 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:14.529154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" event={"ID":"af143b89-583f-4e84-9299-39ce835815c8","Type":"ContainerStarted","Data":"5f5fc2a3a735d5a564ba2091f27f86f769d48c6ef98d7687de273c7d4947e57d"} Jul 13 03:51:14.546676 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:14.546626 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" podStartSLOduration=1.5539262950000001 podStartE2EDuration="2.546610939s" totalImagesPullingTime="992.684644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:51:14.545845503 +0000 UTC m=+579.321803221" watchObservedRunningTime="2026-07-13 03:51:14.546610939 +0000 UTC m=+579.322568634" Jul 13 03:51:21.610806 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:21.610766 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:22.490218 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:22.490188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:22.490404 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:22.490246 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:22.502541 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:22.502520 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:22.577741 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:22.577719 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx64" Jul 13 03:51:34.882496 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:34.882465 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd"] Jul 13 03:51:34.886076 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:34.886050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:34.888077 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:34.888054 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 13 03:51:34.895967 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:34.895939 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd"] Jul 13 03:51:35.012889 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.012857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.013080 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.012902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.013080 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.013009 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.013080 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.013045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.013080 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.013062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zn4p\" (UniqueName: \"kubernetes.io/projected/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-kube-api-access-2zn4p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.013252 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.013143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.113863 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.113827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114044 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114114 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114114 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114213 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114213 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zn4p\" (UniqueName: \"kubernetes.io/projected/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-kube-api-access-2zn4p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114416 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114497 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.114497 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.114469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.116057 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.116034 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.116423 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.116405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.121622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.121598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zn4p\" (UniqueName: \"kubernetes.io/projected/584e23ad-7e7d-4bdd-9a40-2d3db4084b23-kube-api-access-2zn4p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9csjd\" (UID: \"584e23ad-7e7d-4bdd-9a40-2d3db4084b23\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.196700 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.196638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:35.322865 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.322840 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd"] Jul 13 03:51:35.325411 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:51:35.325379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584e23ad_7e7d_4bdd_9a40_2d3db4084b23.slice/crio-88659b8d95e94147e008098e46035cd97e04e252f66086e46793a3566a062560 WatchSource:0}: Error finding container 88659b8d95e94147e008098e46035cd97e04e252f66086e46793a3566a062560: Status 404 returned error can't find the container with id 88659b8d95e94147e008098e46035cd97e04e252f66086e46793a3566a062560 Jul 13 03:51:35.627920 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.627882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" event={"ID":"584e23ad-7e7d-4bdd-9a40-2d3db4084b23","Type":"ContainerStarted","Data":"b165fcfb9cffc1f3fc96cdae09a0e88d8e472f40fa2573cd53a5e224973379ba"} Jul 13 03:51:35.628094 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.627951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" event={"ID":"584e23ad-7e7d-4bdd-9a40-2d3db4084b23","Type":"ContainerStarted","Data":"88659b8d95e94147e008098e46035cd97e04e252f66086e46793a3566a062560"} Jul 13 03:51:35.644392 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.644342 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" podStartSLOduration=1.471154309 podStartE2EDuration="1.644322074s" totalImagesPullingTime="173.167765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:51:35.641697143 +0000 UTC m=+600.417654843" watchObservedRunningTime="2026-07-13 03:51:35.644322074 +0000 UTC m=+600.420279754" Jul 13 03:51:35.652414 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.652386 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:51:35.652803 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:35.652787 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:51:37.003313 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:37.003278 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:44.577746 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.577703 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r"] Jul 13 03:51:44.583566 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.583546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.585410 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.585386 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 13 03:51:44.591117 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.590888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r"] Jul 13 03:51:44.696389 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.696542 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.696542 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.696542 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.696711 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.696711 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.696626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gss8\" (UniqueName: \"kubernetes.io/projected/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-kube-api-access-6gss8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797445 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797406 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797445 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gss8\" (UniqueName: \"kubernetes.io/projected/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-kube-api-access-6gss8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797649 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797649 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797649 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797649 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797912 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797889 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.797957 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.798006 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.797954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.799619 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.799601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.800019 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.799999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.804371 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.804351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gss8\" (UniqueName: \"kubernetes.io/projected/3b4e0ead-5de9-48fe-a585-721bb8e5aac3-kube-api-access-6gss8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r\" (UID: \"3b4e0ead-5de9-48fe-a585-721bb8e5aac3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:44.894491 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:44.894418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:45.025449 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.025410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r"] Jul 13 03:51:45.031884 ip-10-0-133-120 kubenswrapper[2562]: W0713 03:51:45.031854 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4e0ead_5de9_48fe_a585_721bb8e5aac3.slice/crio-3c3fb16b7f8a9e158adf2fd30819e01fa7dddbdfaaf852e0e67387df0ee5671b WatchSource:0}: Error finding container 3c3fb16b7f8a9e158adf2fd30819e01fa7dddbdfaaf852e0e67387df0ee5671b: Status 404 returned error can't find the container with id 3c3fb16b7f8a9e158adf2fd30819e01fa7dddbdfaaf852e0e67387df0ee5671b Jul 13 03:51:45.196941 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.196923 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:45.197024 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.197014 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:45.212267 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.212241 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:45.676415 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.676378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" event={"ID":"3b4e0ead-5de9-48fe-a585-721bb8e5aac3","Type":"ContainerStarted","Data":"b3a3eb49ad315d78374651d29cb043ed8df7110b3876c96df960ea14cb51158e"} Jul 13 03:51:45.676872 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.676422 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" event={"ID":"3b4e0ead-5de9-48fe-a585-721bb8e5aac3","Type":"ContainerStarted","Data":"3c3fb16b7f8a9e158adf2fd30819e01fa7dddbdfaaf852e0e67387df0ee5671b"} Jul 13 03:51:45.687455 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.687431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9csjd" Jul 13 03:51:45.691626 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:45.691585 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" podStartSLOduration=1.5299963490000001 podStartE2EDuration="1.691572045s" totalImagesPullingTime="161.575696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 03:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 03:51:45.690492605 +0000 UTC m=+610.466450351" watchObservedRunningTime="2026-07-13 03:51:45.691572045 +0000 UTC m=+610.467529783" Jul 13 03:51:46.998317 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:46.998286 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:51:54.895055 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:54.895006 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:54.895055 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:54.895067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:54.907385 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:54.907364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:51:55.735266 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:51:55.735231 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r" Jul 13 03:53:04.094704 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:53:04.094666 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:53:13.291851 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:53:13.291811 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:53:24.896330 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:53:24.896295 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:53:32.486891 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:53:32.486812 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:53:43.394037 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:53:43.394004 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:54:52.796181 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:54:52.796143 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:01.893051 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:01.892963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:12.499007 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:12.497821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:21.695501 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:21.695467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:32.290726 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:32.290695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:41.297306 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:41.297268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:55:51.296622 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:55:51.296593 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:56:01.292549 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:01.292512 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:56:23.294372 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:23.294336 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:56:34.495005 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:34.494960 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:56:35.683048 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:35.683015 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:56:35.684328 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:35.684308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 03:56:58.492766 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:56:58.492731 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:57:09.790870 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:57:09.790832 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:57:25.298588 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:57:25.298548 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:57:35.296201 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:57:35.296168 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:57:44.804967 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:57:44.804932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:57:54.400308 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:57:54.400273 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:58:03.391318 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:58:03.391239 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 03:59:52.607741 ip-10-0-133-120 kubenswrapper[2562]: I0713 03:59:52.607701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:03.995601 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:03.995570 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:12.691216 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:12.691181 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:24.600021 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:24.599968 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:35.292756 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:35.292722 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:45.890641 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:45.890608 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:00:55.495022 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:00:55.494972 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:01:06.292277 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:06.292187 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:01:16.988197 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:16.988160 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:01:24.489461 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:24.489425 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:01:33.195318 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:33.195283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:01:35.713144 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:35.713112 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:01:35.715613 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:35.715593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:01:41.497396 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:01:41.497363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:02:00.405554 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:02:00.405519 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:02:09.989063 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:02:09.989034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:02:19.197036 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:02:19.197003 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:02:30.587542 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:02:30.587464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:02:47.695948 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:02:47.695909 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:01.396041 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:01.396011 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:10.488909 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:10.488868 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:23.994372 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:23.994342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:32.988388 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:32.988358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:47.097514 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:47.097478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:03:56.390139 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:03:56.390105 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:05.996964 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:05.996889 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:18.692614 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:18.692578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:27.898702 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:27.898660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:34.693566 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:34.693530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:43.691994 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:43.691943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:04:51.998382 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:04:51.998339 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:03.719061 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:03.719023 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:14.590334 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:14.590297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:24.488176 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:24.488140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:40.598835 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:40.598758 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:50.391084 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:50.391048 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:05:56.892648 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:05:56.892611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:06:19.591847 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:06:19.591810 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:06:28.801400 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:06:28.801361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:06:35.744880 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:06:35.744852 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:06:35.749069 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:06:35.749046 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:07:49.789261 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:07:49.789228 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:07:58.791442 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:07:58.791406 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:08:09.398246 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:08:09.398212 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:08:18.291949 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:08:18.291917 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:08:50.288811 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:08:50.288729 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:09:02.002598 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.002562 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:02.006496 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.006471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.008236 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.008215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c92dd209\"/\"default-dockercfg-zr44v\"" Jul 13 04:09:02.008923 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.008900 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c92dd209\"/\"e2e-lifecycle-model-c92dd209-kserve-self-signed-certs\"" Jul 13 04:09:02.008923 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.008903 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c92dd209\"/\"kube-root-ca.crt\"" Jul 13 04:09:02.009097 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.008927 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c92dd209\"/\"openshift-service-ca.crt\"" Jul 13 04:09:02.014071 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.014038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:02.122960 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.122930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.123087 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.122995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.123087 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.123065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.123159 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.123105 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.123159 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.123132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c666x\" (UniqueName: \"kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.123227 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.123184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.223820 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.223790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.223951 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.223831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.223951 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.223854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224099 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.223964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224099 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.224027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c666x\" (UniqueName: \"kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224099 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.224070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224304 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.224284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224361 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.224320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.224426 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.224406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.226065 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.226050 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.226400 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.226382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.233037 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.233012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c666x\" (UniqueName: \"kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x\") pod \"e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.316798 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.316775 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:02.469454 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.469432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:02.472285 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:09:02.472259 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a66e906_ff6c_4c4b_a800_def42d5a5123.slice/crio-357cc04c9d8724257cfc1e87d6f54f40dce94ee5a5fbfe2610f75de111a205c8 WatchSource:0}: Error finding container 357cc04c9d8724257cfc1e87d6f54f40dce94ee5a5fbfe2610f75de111a205c8: Status 404 returned error can't find the container with id 357cc04c9d8724257cfc1e87d6f54f40dce94ee5a5fbfe2610f75de111a205c8 Jul 13 04:09:02.521031 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:02.521005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" event={"ID":"0a66e906-ff6c-4c4b-a800-def42d5a5123","Type":"ContainerStarted","Data":"357cc04c9d8724257cfc1e87d6f54f40dce94ee5a5fbfe2610f75de111a205c8"} Jul 13 04:09:03.528371 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:03.528335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" event={"ID":"0a66e906-ff6c-4c4b-a800-def42d5a5123","Type":"ContainerStarted","Data":"94e5674683c3e27f6dab6be0d3a39dea6537af7fb3a89b9fca3c9e3515b2a125"} Jul 13 04:09:03.547746 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:03.547692 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" podStartSLOduration=2.547673651 podStartE2EDuration="2.547673651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 04:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:09:03.547628639 +0000 UTC m=+1648.323586341" watchObservedRunningTime="2026-07-13 04:09:03.547673651 +0000 UTC m=+1648.323631355" Jul 13 04:09:05.489574 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:05.489544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:09:12.317790 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:12.317750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:12.317790 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:12.317796 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:12.330076 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:12.330052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:12.581606 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:12.581527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:24.107435 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:24.107402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:09:37.552862 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:37.552828 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:37.993395 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:37.993308 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:09:52.575195 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.575142 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerName="main" containerID="cri-o://94e5674683c3e27f6dab6be0d3a39dea6537af7fb3a89b9fca3c9e3515b2a125" gracePeriod=45 Jul 13 04:09:52.579400 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.579367 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": EOF" Jul 13 04:09:52.770939 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.770902 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerID="94e5674683c3e27f6dab6be0d3a39dea6537af7fb3a89b9fca3c9e3515b2a125" exitCode=0 Jul 13 04:09:52.771086 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.770985 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" event={"ID":"0a66e906-ff6c-4c4b-a800-def42d5a5123","Type":"ContainerDied","Data":"94e5674683c3e27f6dab6be0d3a39dea6537af7fb3a89b9fca3c9e3515b2a125"} Jul 13 04:09:52.819904 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.819881 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:52.961111 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961039 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961111 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961093 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961290 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961117 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c666x\" (UniqueName: \"kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961290 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961163 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961290 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961290 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961221 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir\") pod \"0a66e906-ff6c-4c4b-a800-def42d5a5123\" (UID: \"0a66e906-ff6c-4c4b-a800-def42d5a5123\") " Jul 13 04:09:52.961494 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961421 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:09:52.961494 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961431 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home" (OuterVolumeSpecName: "home") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:09:52.961598 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961531 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:09:52.961709 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961627 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-model-cache\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:52.961709 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961646 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-home\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:52.961709 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.961659 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-tmp-dir\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:52.963330 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.963303 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm" (OuterVolumeSpecName: "dshm") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:09:52.963717 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.963695 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 04:09:52.963940 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:52.963705 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x" (OuterVolumeSpecName: "kube-api-access-c666x") pod "0a66e906-ff6c-4c4b-a800-def42d5a5123" (UID: "0a66e906-ff6c-4c4b-a800-def42d5a5123"). InnerVolumeSpecName "kube-api-access-c666x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 04:09:53.063091 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.063041 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a66e906-ff6c-4c4b-a800-def42d5a5123-dshm\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:53.063091 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.063088 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c666x\" (UniqueName: \"kubernetes.io/projected/0a66e906-ff6c-4c4b-a800-def42d5a5123-kube-api-access-c666x\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:53.063091 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.063098 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a66e906-ff6c-4c4b-a800-def42d5a5123-tls-certs\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:09:53.776432 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.776405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" event={"ID":"0a66e906-ff6c-4c4b-a800-def42d5a5123","Type":"ContainerDied","Data":"357cc04c9d8724257cfc1e87d6f54f40dce94ee5a5fbfe2610f75de111a205c8"} Jul 13 04:09:53.776794 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.776439 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt" Jul 13 04:09:53.776794 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.776450 2562 scope.go:117] "RemoveContainer" containerID="94e5674683c3e27f6dab6be0d3a39dea6537af7fb3a89b9fca3c9e3515b2a125" Jul 13 04:09:53.790188 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.790160 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:53.793042 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:53.793019 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c92dd209/e2e-lifecycle-model-c92dd209-kserve-575d5dbf95-zg5pt"] Jul 13 04:09:55.753474 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:09:55.753443 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" path="/var/lib/kubelet/pods/0a66e906-ff6c-4c4b-a800-def42d5a5123/volumes" Jul 13 04:10:17.191762 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:10:17.191725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:11:23.896249 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:23.896216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:11:34.101938 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:34.101899 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:11:35.780348 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:35.780319 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:11:35.783690 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:35.783670 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:11:38.406607 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.406573 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:11:38.406958 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.406942 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerName="main" Jul 13 04:11:38.406958 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.406953 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerName="main" Jul 13 04:11:38.407072 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.407044 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a66e906-ff6c-4c4b-a800-def42d5a5123" containerName="main" Jul 13 04:11:38.409762 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.409743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.411573 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.411551 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-42c2d3\"/\"openshift-service-ca.crt\"" Jul 13 04:11:38.411699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.411684 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-42c2d3\"/\"kube-root-ca.crt\"" Jul 13 04:11:38.412233 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.412210 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-42c2d3\"/\"test-model-42c2d3-kserve-self-signed-certs\"" Jul 13 04:11:38.412338 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.412247 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-42c2d3\"/\"default-dockercfg-hlhd8\"" Jul 13 04:11:38.419785 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.419767 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:11:38.570274 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.570447 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.570447 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570349 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.570447 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.570598 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.570598 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.570516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cf8q\" (UniqueName: \"kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.671971 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.671897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.671971 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.671944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.671971 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.671962 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672232 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.671997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672232 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.672033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672232 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.672059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cf8q\" (UniqueName: \"kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672380 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.672353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672417 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.672393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.672486 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.672465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.674233 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.674213 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.674475 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.674453 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.679253 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.679235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cf8q\" (UniqueName: \"kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q\") pod \"test-model-42c2d3-kserve-7c5dbdb697-ng5hp\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.719414 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.719391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:38.847884 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:38.847854 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:11:38.850567 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:11:38.850533 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d3cc16b_f3fc_4a6f_801e_3d9867ab1fe2.slice/crio-8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1 WatchSource:0}: Error finding container 8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1: Status 404 returned error can't find the container with id 8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1 Jul 13 04:11:39.280297 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:39.280262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" event={"ID":"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2","Type":"ContainerStarted","Data":"1b7e9acadbeec0550f157acf988be8d0475e5125e054737eb7a75aa5cdbcf8e8"} Jul 13 04:11:39.280297 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:39.280299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" event={"ID":"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2","Type":"ContainerStarted","Data":"8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1"} Jul 13 04:11:39.294543 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:39.294493 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" podStartSLOduration=1.294475934 podStartE2EDuration="1.294475934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 04:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:11:39.29384763 +0000 UTC m=+1804.069805359" watchObservedRunningTime="2026-07-13 04:11:39.294475934 +0000 UTC m=+1804.070433635" Jul 13 04:11:44.192005 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:44.191962 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:11:48.720083 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:48.720038 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:48.720506 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:48.720106 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:48.732744 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:48.732718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:11:49.340907 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:11:49.340878 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:12:16.185080 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.185043 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:16.188694 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.188672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.190684 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.190657 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88358a\"/\"openshift-service-ca.crt\"" Jul 13 04:12:16.191081 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.191064 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88358a\"/\"kube-root-ca.crt\"" Jul 13 04:12:16.191257 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.191238 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88358a\"/\"test-model-88358a-kserve-self-signed-certs\"" Jul 13 04:12:16.191329 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.191281 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88358a\"/\"default-dockercfg-wlqts\"" Jul 13 04:12:16.196741 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.196717 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:16.306112 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.306272 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.306272 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.306272 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whq2w\" (UniqueName: \"kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.306272 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.306489 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.306288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.406778 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.406899 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whq2w\" (UniqueName: \"kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.406899 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.406899 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.406899 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.407146 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.406934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.407289 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.407268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.407352 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.407303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.407352 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.407323 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.409122 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.409106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.409369 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.409351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.415963 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.415945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whq2w\" (UniqueName: \"kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w\") pod \"test-model-88358a-kserve-7cd5556547-zdzxv\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.498799 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.498735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:16.629112 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:16.629088 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:16.631140 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:12:16.631115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd9cc54_cad2_4575_8c26_39336d159b97.slice/crio-96eb6b0c174eedfc5558734c83ae1c1d215a6a0f0355e89e6d37cef3d3ea2cac WatchSource:0}: Error finding container 96eb6b0c174eedfc5558734c83ae1c1d215a6a0f0355e89e6d37cef3d3ea2cac: Status 404 returned error can't find the container with id 96eb6b0c174eedfc5558734c83ae1c1d215a6a0f0355e89e6d37cef3d3ea2cac Jul 13 04:12:17.464409 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:17.464376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" event={"ID":"1dd9cc54-cad2-4575-8c26-39336d159b97","Type":"ContainerStarted","Data":"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68"} Jul 13 04:12:17.464800 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:17.464419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" event={"ID":"1dd9cc54-cad2-4575-8c26-39336d159b97","Type":"ContainerStarted","Data":"96eb6b0c174eedfc5558734c83ae1c1d215a6a0f0355e89e6d37cef3d3ea2cac"} Jul 13 04:12:17.479184 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:17.479133 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" podStartSLOduration=1.479115609 podStartE2EDuration="1.479115609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 04:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:12:17.478105041 +0000 UTC m=+1842.254062742" watchObservedRunningTime="2026-07-13 04:12:17.479115609 +0000 UTC m=+1842.255073304" Jul 13 04:12:20.698686 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:20.698652 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:12:26.499821 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:26.499784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:26.500304 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:26.499833 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:26.512346 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:26.512321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:27.520482 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:27.520451 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:38.368833 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:38.368798 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:12:39.263212 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:39.263184 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:43.894972 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:43.894939 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:12:53.388807 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.388728 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" podUID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" containerName="main" containerID="cri-o://1b7e9acadbeec0550f157acf988be8d0475e5125e054737eb7a75aa5cdbcf8e8" gracePeriod=45 Jul 13 04:12:53.632546 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.632512 2562 generic.go:356] "Generic (PLEG): container finished" podID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" containerID="1b7e9acadbeec0550f157acf988be8d0475e5125e054737eb7a75aa5cdbcf8e8" exitCode=0 Jul 13 04:12:53.632658 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.632563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" event={"ID":"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2","Type":"ContainerDied","Data":"1b7e9acadbeec0550f157acf988be8d0475e5125e054737eb7a75aa5cdbcf8e8"} Jul 13 04:12:53.632658 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.632602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" event={"ID":"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2","Type":"ContainerDied","Data":"8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1"} Jul 13 04:12:53.632658 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.632618 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ded0f830312e700ed6caa731fa9bc9e1bbcbfbfea4aebd375315384505c3ac1" Jul 13 04:12:53.642771 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.642721 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:12:53.738936 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.738910 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739096 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.738965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739096 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739017 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739096 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739047 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739096 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739080 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739319 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cf8q\" (UniqueName: \"kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q\") pod \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\" (UID: \"3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2\") " Jul 13 04:12:53.739319 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739220 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:53.739421 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739345 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home" (OuterVolumeSpecName: "home") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:53.739421 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739375 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:53.739541 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739522 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tmp-dir\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:53.739595 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739544 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-home\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:53.739595 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.739553 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-model-cache\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:53.741311 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.741284 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm" (OuterVolumeSpecName: "dshm") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:53.741311 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.741307 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 04:12:53.741480 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.741391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q" (OuterVolumeSpecName: "kube-api-access-7cf8q") pod "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" (UID: "3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2"). InnerVolumeSpecName "kube-api-access-7cf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 04:12:53.840189 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.840165 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cf8q\" (UniqueName: \"kubernetes.io/projected/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-kube-api-access-7cf8q\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:53.840189 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.840187 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-dshm\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:53.840333 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:53.840197 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2-tls-certs\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.283657 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.283621 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" podUID="1dd9cc54-cad2-4575-8c26-39336d159b97" containerName="main" containerID="cri-o://b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68" gracePeriod=45 Jul 13 04:12:54.535315 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.535254 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:54.638177 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638147 2562 generic.go:356] "Generic (PLEG): container finished" podID="1dd9cc54-cad2-4575-8c26-39336d159b97" containerID="b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68" exitCode=0 Jul 13 04:12:54.638320 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638187 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" Jul 13 04:12:54.638320 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" event={"ID":"1dd9cc54-cad2-4575-8c26-39336d159b97","Type":"ContainerDied","Data":"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68"} Jul 13 04:12:54.638320 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv" event={"ID":"1dd9cc54-cad2-4575-8c26-39336d159b97","Type":"ContainerDied","Data":"96eb6b0c174eedfc5558734c83ae1c1d215a6a0f0355e89e6d37cef3d3ea2cac"} Jul 13 04:12:54.638320 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638293 2562 scope.go:117] "RemoveContainer" containerID="b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68" Jul 13 04:12:54.638320 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.638297 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp" Jul 13 04:12:54.646893 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.646874 2562 scope.go:117] "RemoveContainer" containerID="b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68" Jul 13 04:12:54.647100 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647081 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647184 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647170 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647240 ip-10-0-133-120 kubenswrapper[2562]: E0713 04:12:54.647202 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68\": container with ID starting with b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68 not found: ID does not exist" containerID="b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68" Jul 13 04:12:54.647240 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647240 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647225 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68"} err="failed to get container status \"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68\": rpc error: code = NotFound desc = could not find container \"b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68\": container with ID starting with b9b957e2e9dc2d4d0c385b0d1756502b99307764fe40d8983d2c8f36a811cf68 not found: ID does not exist" Jul 13 04:12:54.647389 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647258 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647389 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647292 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647389 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647312 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home" (OuterVolumeSpecName: "home") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:54.647389 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647332 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whq2w\" (UniqueName: \"kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w\") pod \"1dd9cc54-cad2-4575-8c26-39336d159b97\" (UID: \"1dd9cc54-cad2-4575-8c26-39336d159b97\") " Jul 13 04:12:54.647579 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:54.647579 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647508 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:54.647670 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647654 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-model-cache\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.647705 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647675 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-tmp-dir\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.647705 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.647689 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-home\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.649377 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.649357 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 04:12:54.649477 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.649447 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w" (OuterVolumeSpecName: "kube-api-access-whq2w") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "kube-api-access-whq2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 04:12:54.649521 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.649481 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm" (OuterVolumeSpecName: "dshm") pod "1dd9cc54-cad2-4575-8c26-39336d159b97" (UID: "1dd9cc54-cad2-4575-8c26-39336d159b97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 04:12:54.652340 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.652320 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:12:54.655487 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.655467 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-42c2d3/test-model-42c2d3-kserve-7c5dbdb697-ng5hp"] Jul 13 04:12:54.748948 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.748927 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd9cc54-cad2-4575-8c26-39336d159b97-tls-certs\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.748948 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.748945 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd9cc54-cad2-4575-8c26-39336d159b97-dshm\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.749103 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.748956 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whq2w\" (UniqueName: \"kubernetes.io/projected/1dd9cc54-cad2-4575-8c26-39336d159b97-kube-api-access-whq2w\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:12:54.956968 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.956939 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:54.959006 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:54.958966 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-88358a/test-model-88358a-kserve-7cd5556547-zdzxv"] Jul 13 04:12:55.758994 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:55.758954 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd9cc54-cad2-4575-8c26-39336d159b97" path="/var/lib/kubelet/pods/1dd9cc54-cad2-4575-8c26-39336d159b97/volumes" Jul 13 04:12:55.759528 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:12:55.759506 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" path="/var/lib/kubelet/pods/3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2/volumes" Jul 13 04:13:26.302056 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:13:26.302024 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:13:40.395601 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:13:40.395568 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:13:57.387832 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:13:57.387795 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:14:10.697773 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:14:10.697738 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:14:24.191245 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:14:24.191213 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:15:00.138271 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138189 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138586 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd9cc54-cad2-4575-8c26-39336d159b97" containerName="main" Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138597 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd9cc54-cad2-4575-8c26-39336d159b97" containerName="main" Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138615 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" containerName="main" Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138621 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" containerName="main" Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138683 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3cc16b-f3fc-4a6f-801e-3d9867ab1fe2" containerName="main" Jul 13 04:15:00.138699 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.138691 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd9cc54-cad2-4575-8c26-39336d159b97" containerName="main" Jul 13 04:15:00.141630 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.141611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:15:00.143271 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.143252 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kd7hr\"" Jul 13 04:15:00.143579 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.143561 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 04:15:00.143690 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.143669 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 04:15:00.148650 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.148627 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:15:00.241404 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.241379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfvf\" (UniqueName: \"kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf\") pod \"maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn\" (UID: \"13f2d033-878c-4861-9409-b5bc58c965a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:15:00.341871 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.341839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfvf\" (UniqueName: \"kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf\") pod \"maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn\" (UID: \"13f2d033-878c-4861-9409-b5bc58c965a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:15:00.349800 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.349779 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfvf\" (UniqueName: \"kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf\") pod \"maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn\" (UID: \"13f2d033-878c-4861-9409-b5bc58c965a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:15:00.453439 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.453376 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:15:00.582638 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.582615 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:15:00.585282 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:15:00.585246 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f2d033_878c_4861_9409_b5bc58c965a1.slice/crio-196de63aad3e730a07b1a5ec7525a4790fd7523b22d7e01a44a7b34b99ce72d7 WatchSource:0}: Error finding container 196de63aad3e730a07b1a5ec7525a4790fd7523b22d7e01a44a7b34b99ce72d7: Status 404 returned error can't find the container with id 196de63aad3e730a07b1a5ec7525a4790fd7523b22d7e01a44a7b34b99ce72d7 Jul 13 04:15:00.587138 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:00.587121 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 04:15:01.242582 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:01.242548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" event={"ID":"13f2d033-878c-4861-9409-b5bc58c965a1","Type":"ContainerStarted","Data":"196de63aad3e730a07b1a5ec7525a4790fd7523b22d7e01a44a7b34b99ce72d7"} Jul 13 04:15:04.260620 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:04.260576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" event={"ID":"13f2d033-878c-4861-9409-b5bc58c965a1","Type":"ContainerStarted","Data":"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73"} Jul 13 04:15:04.274959 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:15:04.274896 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" podStartSLOduration=1.575170887 podStartE2EDuration="4.274875593s" totalImagesPullingTime="2.699704706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:15:04.273228148 +0000 UTC m=+2009.049185851" watchObservedRunningTime="2026-07-13 04:15:04.274875593 +0000 UTC m=+2009.050833293" Jul 13 04:16:00.010901 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:00.010808 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:16:00.011517 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:00.011169 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" podUID="13f2d033-878c-4861-9409-b5bc58c965a1" containerName="cleanup" containerID="cri-o://05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73" gracePeriod=30 Jul 13 04:16:30.202687 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.202657 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:16:30.319432 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.319338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wfvf\" (UniqueName: \"kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf\") pod \"13f2d033-878c-4861-9409-b5bc58c965a1\" (UID: \"13f2d033-878c-4861-9409-b5bc58c965a1\") " Jul 13 04:16:30.321509 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.321465 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf" (OuterVolumeSpecName: "kube-api-access-9wfvf") pod "13f2d033-878c-4861-9409-b5bc58c965a1" (UID: "13f2d033-878c-4861-9409-b5bc58c965a1"). InnerVolumeSpecName "kube-api-access-9wfvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 04:16:30.420414 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.420371 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wfvf\" (UniqueName: \"kubernetes.io/projected/13f2d033-878c-4861-9409-b5bc58c965a1-kube-api-access-9wfvf\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\"" Jul 13 04:16:30.702827 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.702719 2562 generic.go:356] "Generic (PLEG): container finished" podID="13f2d033-878c-4861-9409-b5bc58c965a1" containerID="05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73" exitCode=137 Jul 13 04:16:30.702827 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.702769 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" Jul 13 04:16:30.702827 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.702811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" event={"ID":"13f2d033-878c-4861-9409-b5bc58c965a1","Type":"ContainerDied","Data":"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73"} Jul 13 04:16:30.703136 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.702877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn" event={"ID":"13f2d033-878c-4861-9409-b5bc58c965a1","Type":"ContainerDied","Data":"196de63aad3e730a07b1a5ec7525a4790fd7523b22d7e01a44a7b34b99ce72d7"} Jul 13 04:16:30.703136 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.702908 2562 scope.go:117] "RemoveContainer" containerID="05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73" Jul 13 04:16:30.713129 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.713109 2562 scope.go:117] "RemoveContainer" containerID="05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73" Jul 13 04:16:30.713422 ip-10-0-133-120 kubenswrapper[2562]: E0713 04:16:30.713403 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73\": container with ID starting with 05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73 not found: ID does not exist" containerID="05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73" Jul 13 04:16:30.713468 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.713433 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73"} err="failed to get container status \"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73\": rpc error: code = NotFound desc = could not find container \"05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73\": container with ID starting with 05ed5c0c1a34781c8dff81875d5b8a0c32d947ab19df953c2f2edd1b2a345b73 not found: ID does not exist" Jul 13 04:16:30.723345 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.723310 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:16:30.728048 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:30.728012 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-830ad7-29731935-9swcn"] Jul 13 04:16:31.753048 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:31.753012 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f2d033-878c-4861-9409-b5bc58c965a1" path="/var/lib/kubelet/pods/13f2d033-878c-4861-9409-b5bc58c965a1/volumes" Jul 13 04:16:35.815560 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:35.815517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:16:35.819269 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:16:35.819245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5xf7_7cf560ba-63be-4f6d-a5f9-5af6100690d0/ovn-acl-logging/0.log" Jul 13 04:17:31.398208 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:31.398118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:17:42.492931 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:42.492894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hfffn"] Jul 13 04:17:46.596167 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:46.596138 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6757d97b-xkwsj_9c242bb3-eb92-400a-bbd3-97579b45a795/copy-manifests/0.log" Jul 13 04:17:46.603348 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:46.603322 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6757d97b-xkwsj_9c242bb3-eb92-400a-bbd3-97579b45a795/manager/0.log" Jul 13 04:17:48.076458 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:48.076418 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jr9w5_00f68645-ffb7-4117-968b-3ab8b4d2886d/manager/0.log" Jul 13 04:17:48.188721 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:48.188694 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9ghzd_ec92b4be-4f15-4aa2-8901-3113a1458f6a/kuadrant-console-plugin/0.log" Jul 13 04:17:48.540492 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:48.540468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hfffn_9b88b8bc-fb3a-4792-99c1-0b44dba5b96d/limitador/0.log" Jul 13 04:17:49.222839 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:49.222811 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bfd7dd56-9qsvx_2409f770-75df-4e5b-8832-0fba80212182/kube-auth-proxy/0.log" Jul 13 04:17:50.362057 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:50.362027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tsx64_af143b89-583f-4e84-9299-39ce835815c8/main/0.log" Jul 13 04:17:50.587245 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:50.587200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9csjd_584e23ad-7e7d-4bdd-9a40-2d3db4084b23/main/0.log" Jul 13 04:17:50.701524 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:50.701438 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-s927r_3b4e0ead-5de9-48fe-a585-721bb8e5aac3/main/0.log" Jul 13 04:17:55.075840 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.075796 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xpvgg/must-gather-b4sg8"] Jul 13 04:17:55.078262 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.076241 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f2d033-878c-4861-9409-b5bc58c965a1" containerName="cleanup" Jul 13 04:17:55.078262 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.076253 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f2d033-878c-4861-9409-b5bc58c965a1" containerName="cleanup" Jul 13 04:17:55.078262 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.076324 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f2d033-878c-4861-9409-b5bc58c965a1" containerName="cleanup" Jul 13 04:17:55.079445 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.079419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.081552 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.081521 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xpvgg\"/\"openshift-service-ca.crt\"" Jul 13 04:17:55.081730 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.081600 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xpvgg\"/\"default-dockercfg-k7b9k\"" Jul 13 04:17:55.081900 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.081877 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xpvgg\"/\"kube-root-ca.crt\"" Jul 13 04:17:55.087043 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.087008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xpvgg/must-gather-b4sg8"] Jul 13 04:17:55.145125 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.145076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6035fb-e190-4dcd-b49c-9108efb61631-must-gather-output\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.145311 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.145146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2h8\" (UniqueName: \"kubernetes.io/projected/cd6035fb-e190-4dcd-b49c-9108efb61631-kube-api-access-hg2h8\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.246426 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.246387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6035fb-e190-4dcd-b49c-9108efb61631-must-gather-output\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.246610 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.246453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2h8\" (UniqueName: \"kubernetes.io/projected/cd6035fb-e190-4dcd-b49c-9108efb61631-kube-api-access-hg2h8\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.246774 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.246755 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd6035fb-e190-4dcd-b49c-9108efb61631-must-gather-output\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.254291 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.254255 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2h8\" (UniqueName: \"kubernetes.io/projected/cd6035fb-e190-4dcd-b49c-9108efb61631-kube-api-access-hg2h8\") pod \"must-gather-b4sg8\" (UID: \"cd6035fb-e190-4dcd-b49c-9108efb61631\") " pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.391209 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.391110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" Jul 13 04:17:55.531691 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:55.531588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xpvgg/must-gather-b4sg8"] Jul 13 04:17:55.534818 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:17:55.534781 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6035fb_e190_4dcd_b49c_9108efb61631.slice/crio-4c58b4466cfcefaa2d2d520dadbc573fda7987b6e73e29ed6881e546cedb5166 WatchSource:0}: Error finding container 4c58b4466cfcefaa2d2d520dadbc573fda7987b6e73e29ed6881e546cedb5166: Status 404 returned error can't find the container with id 4c58b4466cfcefaa2d2d520dadbc573fda7987b6e73e29ed6881e546cedb5166 Jul 13 04:17:56.140775 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:56.140734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" event={"ID":"cd6035fb-e190-4dcd-b49c-9108efb61631","Type":"ContainerStarted","Data":"4c58b4466cfcefaa2d2d520dadbc573fda7987b6e73e29ed6881e546cedb5166"} Jul 13 04:17:57.150429 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:57.150110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" event={"ID":"cd6035fb-e190-4dcd-b49c-9108efb61631","Type":"ContainerStarted","Data":"ca7c0475c7e56c454504678e0bbe178d59218e03c608427b8be46bc334d78e74"} Jul 13 04:17:57.150429 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:57.150189 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" event={"ID":"cd6035fb-e190-4dcd-b49c-9108efb61631","Type":"ContainerStarted","Data":"e4b738a50357e1e6e10ecd779482f0b57fdadcf1cc530140378a2a739168dda0"} Jul 13 04:17:57.165676 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:57.165602 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xpvgg/must-gather-b4sg8" podStartSLOduration=1.181436957 podStartE2EDuration="2.165576122s" totalImagesPullingTime="984.139165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 04:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:17:57.163466783 +0000 UTC m=+2181.939424484" watchObservedRunningTime="2026-07-13 04:17:57.165576122 +0000 UTC m=+2181.941533823" Jul 13 04:17:58.076241 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:58.076200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-45z9p_dbe98aea-97ba-462c-b92d-4d147e56fd4d/global-pull-secret-syncer/0.log" Jul 13 04:17:58.268533 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:58.268491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tjvh9_db23e086-5d21-4d81-9396-72be63d48a33/konnectivity-agent/0.log" Jul 13 04:17:58.294452 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:17:58.294390 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-120.ec2.internal_7aa2a3eea528d01b7f1c87c2d7e7e408/haproxy/0.log" Jul 13 04:18:03.699551 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:03.699518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jr9w5_00f68645-ffb7-4117-968b-3ab8b4d2886d/manager/0.log" Jul 13 04:18:03.776928 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:03.776894 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9ghzd_ec92b4be-4f15-4aa2-8901-3113a1458f6a/kuadrant-console-plugin/0.log" Jul 13 04:18:03.957157 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:03.957128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hfffn_9b88b8bc-fb3a-4792-99c1-0b44dba5b96d/limitador/0.log" Jul 13 04:18:05.891067 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:05.891030 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/alertmanager/0.log" Jul 13 04:18:05.916660 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:05.916622 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/config-reloader/0.log" Jul 13 04:18:05.943165 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:05.943071 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/kube-rbac-proxy-web/0.log" Jul 13 04:18:05.969918 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:05.969873 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/kube-rbac-proxy/0.log" Jul 13 04:18:05.991627 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:05.991597 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/kube-rbac-proxy-metric/0.log" Jul 13 04:18:06.017503 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.017408 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/prom-label-proxy/0.log" Jul 13 04:18:06.039272 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.039244 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f7affbe-c560-4b02-b519-15687361374b/init-config-reloader/0.log" Jul 13 04:18:06.100530 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.100487 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xr84f_b9f708cc-5ab0-4823-b46f-a34e9c739683/kube-state-metrics/0.log" Jul 13 04:18:06.117929 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.117899 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xr84f_b9f708cc-5ab0-4823-b46f-a34e9c739683/kube-rbac-proxy-main/0.log" Jul 13 04:18:06.139086 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.139055 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xr84f_b9f708cc-5ab0-4823-b46f-a34e9c739683/kube-rbac-proxy-self/0.log" Jul 13 04:18:06.165236 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.165137 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c7d55bcc7-hqzsh_1b8deeb2-a2e1-4fdf-a2fb-f4d2e58b4dc4/metrics-server/0.log" Jul 13 04:18:06.195691 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.195650 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-dk4zz_c57e8fba-17c6-47fb-bc01-3d01d2c64e32/monitoring-plugin/0.log" Jul 13 04:18:06.388871 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.388833 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5t_443e8e18-faf1-409b-9461-49bc5e1445f5/node-exporter/0.log" Jul 13 04:18:06.408856 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.408822 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5t_443e8e18-faf1-409b-9461-49bc5e1445f5/kube-rbac-proxy/0.log" Jul 13 04:18:06.431511 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.431413 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjb5t_443e8e18-faf1-409b-9461-49bc5e1445f5/init-textfile/0.log" Jul 13 04:18:06.458825 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.458777 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q2jt5_ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1/kube-rbac-proxy-main/0.log" Jul 13 04:18:06.480483 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.480456 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q2jt5_ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1/kube-rbac-proxy-self/0.log" Jul 13 04:18:06.503321 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.503295 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q2jt5_ef39cb7f-a940-4ce5-ab54-bdcbe47ab4f1/openshift-state-metrics/0.log" Jul 13 04:18:06.544529 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.544486 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/prometheus/0.log" Jul 13 04:18:06.568117 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.568086 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/config-reloader/0.log" Jul 13 04:18:06.589664 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.589628 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/thanos-sidecar/0.log" Jul 13 04:18:06.612318 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.612286 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/kube-rbac-proxy-web/0.log" Jul 13 04:18:06.635235 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.635204 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/kube-rbac-proxy/0.log" Jul 13 04:18:06.656818 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.656786 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/kube-rbac-proxy-thanos/0.log" Jul 13 04:18:06.678153 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.678125 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d86a4766-d08a-41c2-a1ec-5da4ac6143b0/init-config-reloader/0.log" Jul 13 04:18:06.705823 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.705688 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6xltc_c26d586b-9e78-4739-9b47-7600717c3298/prometheus-operator/0.log" Jul 13 04:18:06.726120 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.726085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6xltc_c26d586b-9e78-4739-9b47-7600717c3298/kube-rbac-proxy/0.log" Jul 13 04:18:06.790041 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.790009 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7f5f8897-xmr8b_e531e19e-0200-48ff-b34d-e864cb3e01ea/telemeter-client/0.log" Jul 13 04:18:06.810969 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.810939 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7f5f8897-xmr8b_e531e19e-0200-48ff-b34d-e864cb3e01ea/reload/0.log" Jul 13 04:18:06.841590 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.841558 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7f5f8897-xmr8b_e531e19e-0200-48ff-b34d-e864cb3e01ea/kube-rbac-proxy/0.log" Jul 13 04:18:06.866461 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.866436 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/thanos-query/0.log" Jul 13 04:18:06.888133 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.888105 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/kube-rbac-proxy-web/0.log" Jul 13 04:18:06.912487 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.912389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/kube-rbac-proxy/0.log" Jul 13 04:18:06.932518 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.932487 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/prom-label-proxy/0.log" Jul 13 04:18:06.954118 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.954080 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/kube-rbac-proxy-rules/0.log" Jul 13 04:18:06.977682 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:06.977588 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7bbf66d4-tcqnm_babf3c3d-8784-47f4-9aa2-ffe8b3f3a40e/kube-rbac-proxy-metrics/0.log" Jul 13 04:18:08.570739 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.570695 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h"] Jul 13 04:18:08.577598 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.577565 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.582378 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.582342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h"] Jul 13 04:18:08.601484 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.601446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-lib-modules\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.601708 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.601681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-proc\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.601803 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.601768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-sys\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.601869 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.601825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-podres\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.601921 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.601864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnd6\" (UniqueName: \"kubernetes.io/projected/e7760df4-0120-46c5-8e3c-4fb6becb525c-kube-api-access-pjnd6\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnd6\" (UniqueName: \"kubernetes.io/projected/e7760df4-0120-46c5-8e3c-4fb6becb525c-kube-api-access-pjnd6\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-lib-modules\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-proc\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-sys\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-podres\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.703931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-podres\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.704440 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-lib-modules\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.704508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-proc\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.704584 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.704556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7760df4-0120-46c5-8e3c-4fb6becb525c-sys\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.712616 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.712574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnd6\" (UniqueName: \"kubernetes.io/projected/e7760df4-0120-46c5-8e3c-4fb6becb525c-kube-api-access-pjnd6\") pod \"perf-node-gather-daemonset-rg62h\" (UID: \"e7760df4-0120-46c5-8e3c-4fb6becb525c\") " pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:08.893590 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:08.893484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:09.081426 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:09.081297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h"] Jul 13 04:18:09.084698 ip-10-0-133-120 kubenswrapper[2562]: W0713 04:18:09.084652 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7760df4_0120_46c5_8e3c_4fb6becb525c.slice/crio-b67d2f0906a252e5b28eae7c6b086aa848def01b047716e10168061dbdcd9e0d WatchSource:0}: Error finding container b67d2f0906a252e5b28eae7c6b086aa848def01b047716e10168061dbdcd9e0d: Status 404 returned error can't find the container with id b67d2f0906a252e5b28eae7c6b086aa848def01b047716e10168061dbdcd9e0d Jul 13 04:18:09.252166 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:09.252120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" event={"ID":"e7760df4-0120-46c5-8e3c-4fb6becb525c","Type":"ContainerStarted","Data":"8d050239fd71408ece13c293de7b3c09e747784e64311f6593525d236b13d795"} Jul 13 04:18:09.252350 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:09.252195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" event={"ID":"e7760df4-0120-46c5-8e3c-4fb6becb525c","Type":"ContainerStarted","Data":"b67d2f0906a252e5b28eae7c6b086aa848def01b047716e10168061dbdcd9e0d"} Jul 13 04:18:09.252720 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:09.252696 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:11.056254 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:11.056218 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54vc8_643c3520-aed7-426c-861c-82bb1830bebf/dns/0.log" Jul 13 04:18:11.077027 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:11.076962 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54vc8_643c3520-aed7-426c-861c-82bb1830bebf/kube-rbac-proxy/0.log" Jul 13 04:18:11.237645 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:11.237620 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wpr6r_e93eb64b-9ff7-4ada-ad80-286e166db78a/dns-node-resolver/0.log" Jul 13 04:18:11.759305 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:11.759266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mg78z_55459b3f-e007-4218-9cf8-85f8f7bb9976/node-ca/0.log" Jul 13 04:18:12.848737 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:12.848705 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bfd7dd56-9qsvx_2409f770-75df-4e5b-8832-0fba80212182/kube-auth-proxy/0.log" Jul 13 04:18:13.689472 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:13.689442 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8v5qz_c1b63269-4ba5-4d49-9026-74118fbc5c23/serve-healthcheck-canary/0.log" Jul 13 04:18:14.216318 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:14.216291 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcc4t_bd24fd38-323b-4b4d-89cd-48a3a99bbfe6/kube-rbac-proxy/0.log" Jul 13 04:18:14.237238 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:14.237212 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcc4t_bd24fd38-323b-4b4d-89cd-48a3a99bbfe6/exporter/0.log" Jul 13 04:18:14.258146 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:14.258103 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcc4t_bd24fd38-323b-4b4d-89cd-48a3a99bbfe6/extractor/0.log" Jul 13 04:18:15.270876 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:15.270836 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" Jul 13 04:18:15.284952 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:15.284885 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xpvgg/perf-node-gather-daemonset-rg62h" podStartSLOduration=7.284863777 podStartE2EDuration="7.284863777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 04:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 04:18:09.265721031 +0000 UTC m=+2194.041678759" watchObservedRunningTime="2026-07-13 04:18:15.284863777 +0000 UTC m=+2200.060821478" Jul 13 04:18:17.150633 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:17.150513 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6757d97b-xkwsj_9c242bb3-eb92-400a-bbd3-97579b45a795/manager/0.log" Jul 13 04:18:17.174816 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:17.174781 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6757d97b-xkwsj_9c242bb3-eb92-400a-bbd3-97579b45a795/copy-manifests/0.log" Jul 13 04:18:18.445454 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:18.445418 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c9556dbfd-9r4f4_cf0fb43e-d0f5-4454-b995-47864bcf0927/manager/0.log" Jul 13 04:18:20.050907 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:20.050871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bfd7dd56-9qsvx_2409f770-75df-4e5b-8832-0fba80212182/kube-auth-proxy/0.log" Jul 13 04:18:21.607480 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:21.607448 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bfd7dd56-9qsvx_2409f770-75df-4e5b-8832-0fba80212182/kube-auth-proxy/0.log" Jul 13 04:18:25.272953 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.272872 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/kube-multus-additional-cni-plugins/0.log" Jul 13 04:18:25.317766 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.317725 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/egress-router-binary-copy/0.log" Jul 13 04:18:25.356755 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.356712 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/cni-plugins/0.log" Jul 13 04:18:25.392906 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.392876 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/bond-cni-plugin/0.log" Jul 13 04:18:25.432840 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.432808 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/routeoverride-cni/0.log" Jul 13 04:18:25.471681 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.471635 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/whereabouts-cni-bincopy/0.log" Jul 13 04:18:25.503541 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.503509 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jdnr6_4fba8a3f-e72d-4de0-9c38-a4c9830ff7fd/whereabouts-cni/0.log" Jul 13 04:18:25.697203 ip-10-0-133-120 kubenswrapper[2562]: I0713 04:18:25.697115 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cq9vg_e22f94d2-04ca-470e-8664-5c33f1ab1691/kube-multus/0.log"