I0423 21:52:10.931680 1 cmd.go:253] Using service-serving-cert provided certificates I0423 21:52:10.932111 1 observer_polling.go:159] Starting file observer I0423 21:52:11.172210 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0423 21:52:11.172564 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0423 21:52:11.172958 1 secure_serving.go:57] Forcing use of http/1.1 only W0423 21:52:11.172978 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0423 21:52:11.172982 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0423 21:52:11.172986 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0423 21:52:11.172990 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0423 21:52:11.172994 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0423 21:52:11.172996 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0423 21:52:11.173504 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0423 21:52:11.178345 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0423 21:52:11.178407 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"bb606198-daf8-4906-b540-43a35eb0f2de", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0423 21:52:11.179255 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0423 21:52:11.179271 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0423 21:52:11.179268 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0423 21:52:11.179281 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 21:52:11.179281 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0423 21:52:11.179304 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 21:52:11.179571 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0423 21:52:11.180337 1 secure_serving.go:213] Serving securely on [::]:8443 I0423 21:52:11.180436 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0423 21:52:11.189510 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0423 21:52:11.189542 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0423 21:52:11.189636 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0423 21:52:11.195506 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0423 21:52:11.195523 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0423 21:52:11.201098 1 secretconfigobserver.go:119] support secret does not exist I0423 21:52:11.206182 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0423 21:52:11.210439 1 secretconfigobserver.go:119] support secret does not exist I0423 21:52:11.213022 1 recorder.go:176] Pruning old reports every 5h8m54s, max age is 288h0m0s I0423 21:52:11.218441 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0423 21:52:11.218457 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0423 21:52:11.218460 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0423 21:52:11.218465 1 insightsreport.go:296] Starting report retriever I0423 21:52:11.218443 1 periodic.go:216] Running clusterconfig gatherer I0423 21:52:11.218472 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0423 21:52:11.218511 1 tasks_processing.go:45] number of workers: 64 I0423 21:52:11.218535 1 tasks_processing.go:69] worker 2 listening for tasks. I0423 21:52:11.218545 1 tasks_processing.go:69] worker 9 listening for tasks. I0423 21:52:11.218547 1 tasks_processing.go:69] worker 0 listening for tasks. I0423 21:52:11.218553 1 tasks_processing.go:69] worker 1 listening for tasks. I0423 21:52:11.218548 1 tasks_processing.go:69] worker 23 listening for tasks. I0423 21:52:11.218553 1 tasks_processing.go:69] worker 10 listening for tasks. I0423 21:52:11.218562 1 tasks_processing.go:69] worker 6 listening for tasks. I0423 21:52:11.218566 1 tasks_processing.go:69] worker 12 listening for tasks. I0423 21:52:11.218569 1 tasks_processing.go:69] worker 5 listening for tasks. I0423 21:52:11.218575 1 tasks_processing.go:69] worker 24 listening for tasks. I0423 21:52:11.218577 1 tasks_processing.go:69] worker 8 listening for tasks. I0423 21:52:11.218583 1 tasks_processing.go:69] worker 14 listening for tasks. I0423 21:52:11.218585 1 tasks_processing.go:71] worker 8 working on machine_config_pools task. I0423 21:52:11.218579 1 tasks_processing.go:69] worker 3 listening for tasks. I0423 21:52:11.218591 1 tasks_processing.go:69] worker 15 listening for tasks. I0423 21:52:11.218571 1 tasks_processing.go:71] worker 6 working on machine_sets task. I0423 21:52:11.218597 1 tasks_processing.go:69] worker 20 listening for tasks. I0423 21:52:11.218596 1 tasks_processing.go:69] worker 48 listening for tasks. I0423 21:52:11.218579 1 tasks_processing.go:71] worker 5 working on service_accounts task. I0423 21:52:11.218604 1 tasks_processing.go:69] worker 49 listening for tasks. I0423 21:52:11.218599 1 tasks_processing.go:69] worker 4 listening for tasks. I0423 21:52:11.218606 1 tasks_processing.go:69] worker 17 listening for tasks. I0423 21:52:11.218615 1 tasks_processing.go:69] worker 50 listening for tasks. I0423 21:52:11.218578 1 tasks_processing.go:69] worker 13 listening for tasks. I0423 21:52:11.218620 1 tasks_processing.go:69] worker 22 listening for tasks. I0423 21:52:11.218561 1 tasks_processing.go:69] worker 11 listening for tasks. I0423 21:52:11.218624 1 tasks_processing.go:69] worker 21 listening for tasks. I0423 21:52:11.218578 1 tasks_processing.go:69] worker 46 listening for tasks. I0423 21:52:11.218585 1 tasks_processing.go:71] worker 24 working on image_pruners task. I0423 21:52:11.218624 1 tasks_processing.go:69] worker 36 listening for tasks. I0423 21:52:11.218638 1 tasks_processing.go:69] worker 53 listening for tasks. I0423 21:52:11.218638 1 tasks_processing.go:69] worker 43 listening for tasks. I0423 21:52:11.218637 1 tasks_processing.go:69] worker 32 listening for tasks. I0423 21:52:11.218650 1 tasks_processing.go:69] worker 41 listening for tasks. I0423 21:52:11.218653 1 tasks_processing.go:69] worker 35 listening for tasks. I0423 21:52:11.218592 1 tasks_processing.go:69] worker 26 listening for tasks. I0423 21:52:11.218652 1 tasks_processing.go:69] worker 34 listening for tasks. I0423 21:52:11.218660 1 tasks_processing.go:69] worker 30 listening for tasks. I0423 21:52:11.218662 1 tasks_processing.go:69] worker 28 listening for tasks. I0423 21:52:11.218662 1 tasks_processing.go:69] worker 56 listening for tasks. I0423 21:52:11.218667 1 tasks_processing.go:71] worker 0 working on openstack_dataplanedeployments task. I0423 21:52:11.218671 1 tasks_processing.go:71] worker 9 working on image task. I0423 21:52:11.218668 1 tasks_processing.go:71] worker 1 working on crds task. I0423 21:52:11.218674 1 tasks_processing.go:71] worker 2 working on jaegers task. I0423 21:52:11.218678 1 tasks_processing.go:69] worker 57 listening for tasks. I0423 21:52:11.218674 1 tasks_processing.go:69] worker 29 listening for tasks. I0423 21:52:11.218685 1 tasks_processing.go:69] worker 60 listening for tasks. I0423 21:52:11.218594 1 tasks_processing.go:69] worker 19 listening for tasks. I0423 21:52:11.218597 1 tasks_processing.go:69] worker 27 listening for tasks. I0423 21:52:11.218695 1 tasks_processing.go:69] worker 61 listening for tasks. I0423 21:52:11.218697 1 tasks_processing.go:69] worker 59 listening for tasks. I0423 21:52:11.218566 1 tasks_processing.go:71] worker 10 working on oauths task. I0423 21:52:11.218650 1 tasks_processing.go:69] worker 31 listening for tasks. I0423 21:52:11.218600 1 tasks_processing.go:69] worker 16 listening for tasks. I0423 21:52:11.218603 1 tasks_processing.go:69] worker 37 listening for tasks. I0423 21:52:11.218607 1 tasks_processing.go:69] worker 38 listening for tasks. I0423 21:52:11.218571 1 tasks_processing.go:71] worker 12 working on machine_healthchecks task. I0423 21:52:11.218589 1 tasks_processing.go:69] worker 47 listening for tasks. I0423 21:52:11.218584 1 tasks_processing.go:69] worker 25 listening for tasks. I0423 21:52:11.218613 1 tasks_processing.go:69] worker 18 listening for tasks. I0423 21:52:11.218572 1 tasks_processing.go:69] worker 7 listening for tasks. I0423 21:52:11.218621 1 tasks_processing.go:69] worker 40 listening for tasks. I0423 21:52:11.218626 1 tasks_processing.go:69] worker 51 listening for tasks. I0423 21:52:11.218629 1 tasks_processing.go:69] worker 44 listening for tasks. I0423 21:52:11.218630 1 tasks_processing.go:69] worker 33 listening for tasks. I0423 21:52:11.218632 1 tasks_processing.go:69] worker 52 listening for tasks. I0423 21:52:11.218794 1 tasks_processing.go:71] worker 52 working on olm_operators task. I0423 21:52:11.218796 1 tasks_processing.go:71] worker 34 working on sap_config task. I0423 21:52:11.218637 1 tasks_processing.go:69] worker 45 listening for tasks. I0423 21:52:11.218646 1 tasks_processing.go:69] worker 42 listening for tasks. I0423 21:52:11.218648 1 tasks_processing.go:69] worker 54 listening for tasks. I0423 21:52:11.218654 1 tasks_processing.go:69] worker 55 listening for tasks. I0423 21:52:11.218672 1 tasks_processing.go:69] worker 63 listening for tasks. I0423 21:52:11.218858 1 tasks_processing.go:71] worker 22 working on openshift_machine_api_events task. I0423 21:52:11.218866 1 tasks_processing.go:71] worker 53 working on aggregated_monitoring_cr_names task. I0423 21:52:11.218613 1 tasks_processing.go:69] worker 39 listening for tasks. I0423 21:52:11.218566 1 tasks_processing.go:71] worker 23 working on openshift_logging task. I0423 21:52:11.219132 1 tasks_processing.go:69] worker 58 listening for tasks. I0423 21:52:11.219147 1 tasks_processing.go:71] worker 29 working on sap_datahubs task. I0423 21:52:11.219192 1 tasks_processing.go:71] worker 33 working on metrics task. I0423 21:52:11.219309 1 tasks_processing.go:71] worker 21 working on monitoring_persistent_volumes task. I0423 21:52:11.219341 1 tasks_processing.go:71] worker 11 working on image_registries task. I0423 21:52:11.219365 1 tasks_processing.go:71] worker 46 working on silenced_alerts task. I0423 21:52:11.219391 1 tasks_processing.go:71] worker 41 working on authentication task. I0423 21:52:11.219405 1 tasks_processing.go:71] worker 43 working on pdbs task. I0423 21:52:11.219432 1 tasks_processing.go:71] worker 32 working on active_alerts task. I0423 21:52:11.219450 1 tasks_processing.go:71] worker 35 working on nodenetworkstates task. I0423 21:52:11.219462 1 tasks_processing.go:71] worker 26 working on certificate_signing_requests task. I0423 21:52:11.220224 1 tasks_processing.go:71] worker 60 working on infrastructures task. I0423 21:52:11.220319 1 tasks_processing.go:71] worker 31 working on ceph_cluster task. I0423 21:52:11.220576 1 tasks_processing.go:71] worker 19 working on proxies task. I0423 21:52:11.220626 1 tasks_processing.go:71] worker 28 working on support_secret task. I0423 21:52:11.220974 1 tasks_processing.go:71] worker 51 working on storage_cluster task. I0423 21:52:11.221038 1 tasks_processing.go:71] worker 50 working on container_runtime_configs task. I0423 21:52:11.218679 1 tasks_processing.go:69] worker 62 listening for tasks. I0423 21:52:11.220654 1 tasks_processing.go:71] worker 3 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0423 21:52:11.221100 1 tasks_processing.go:71] worker 45 working on version task. I0423 21:52:11.221221 1 tasks_processing.go:71] worker 59 working on operators task. I0423 21:52:11.221328 1 tasks_processing.go:71] worker 42 working on tsdb_status task. I0423 21:52:11.220983 1 tasks_processing.go:71] worker 57 working on machine_autoscalers task. I0423 21:52:11.220657 1 tasks_processing.go:71] worker 48 working on nodes task. I0423 21:52:11.221421 1 tasks_processing.go:71] worker 63 working on clusterroles task. I0423 21:52:11.220648 1 tasks_processing.go:71] worker 7 working on openstack_version task. I0423 21:52:11.221101 1 tasks_processing.go:71] worker 62 working on operators_pods_and_events task. I0423 21:52:11.220679 1 tasks_processing.go:71] worker 61 working on mutating_webhook_configurations task. I0423 21:52:11.220717 1 tasks_processing.go:71] worker 15 working on machines task. I0423 21:52:11.220722 1 tasks_processing.go:71] worker 36 working on nodenetworkconfigurationpolicies task. I0423 21:52:11.220797 1 tasks_processing.go:71] worker 17 working on openstack_controlplanes task. I0423 21:52:11.221592 1 tasks_processing.go:71] worker 55 working on ingress task. I0423 21:52:11.220747 1 tasks_processing.go:71] worker 27 working on storage_classes task. I0423 21:52:11.220810 1 tasks_processing.go:71] worker 49 working on networks task. I0423 21:52:11.220807 1 tasks_processing.go:71] worker 56 working on node_features task. I0423 21:52:11.220836 1 tasks_processing.go:71] worker 37 working on validating_webhook_configurations task. I0423 21:52:11.220817 1 tasks_processing.go:71] worker 14 working on lokistack task. I0423 21:52:11.220858 1 tasks_processing.go:71] worker 44 working on config_maps task. I0423 21:52:11.220888 1 tasks_processing.go:71] worker 40 working on machine_configs task. I0423 21:52:11.220899 1 tasks_processing.go:71] worker 18 working on node_logs task. I0423 21:52:11.220904 1 tasks_processing.go:71] worker 38 working on openstack_dataplanenodesets task. I0423 21:52:11.220919 1 tasks_processing.go:71] worker 25 working on sap_pods task. I0423 21:52:11.220925 1 tasks_processing.go:71] worker 4 working on feature_gates task. I0423 21:52:11.221007 1 tasks_processing.go:71] worker 13 working on cost_management_metrics_configs task. I0423 21:52:11.222346 1 tasks_processing.go:71] worker 58 working on overlapping_namespace_uids task. I0423 21:52:11.221102 1 tasks_processing.go:71] worker 30 working on dvo_metrics task. I0423 21:52:11.221432 1 tasks_processing.go:71] worker 54 working on schedulers task. I0423 21:52:11.222561 1 tasks_processing.go:71] worker 39 working on install_plans task. I0423 21:52:11.220717 1 tasks_processing.go:71] worker 16 working on qemu_kubevirt_launcher_logs task. I0423 21:52:11.220735 1 tasks_processing.go:71] worker 20 working on pod_network_connectivity_checks task. I0423 21:52:11.222928 1 tasks_processing.go:71] worker 8 working on ingress_certificates task. I0423 21:52:11.222934 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 4.334238ms to process 0 records I0423 21:52:11.220841 1 tasks_processing.go:71] worker 47 working on cluster_apiserver task. I0423 21:52:11.224204 1 tasks_processing.go:71] worker 29 working on container_images task. I0423 21:52:11.224213 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 5.045331ms to process 0 records I0423 21:52:11.225407 1 tasks_processing.go:74] worker 6 stopped. I0423 21:52:11.225420 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 6.805542ms to process 0 records I0423 21:52:11.225437 1 tasks_processing.go:74] worker 23 stopped. I0423 21:52:11.225442 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 6.502328ms to process 0 records I0423 21:52:11.225732 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0423 21:52:11.225750 1 controller.go:368] The initial operator extension status is healthy I0423 21:52:11.225757 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0423 21:52:11.225762 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0423 21:52:11.225765 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0423 21:52:11.225779 1 controller.go:531] The operator is still being initialized I0423 21:52:11.225785 1 controller.go:554] The operator is healthy I0423 21:52:11.225851 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0423 21:52:11.225866 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0423 21:52:11.225964 1 operator.go:328] started I0423 21:52:11.225983 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0423 21:52:11.225991 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.19.239:9091: connect: connection refused I0423 21:52:11.226012 1 tasks_processing.go:74] worker 42 stopped. E0423 21:52:11.226020 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.19.239:9091: connect: connection refused I0423 21:52:11.226029 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.668207ms to process 0 records E0423 21:52:11.226065 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.19.239:9091: connect: connection refused I0423 21:52:11.226093 1 tasks_processing.go:74] worker 33 stopped. E0423 21:52:11.226104 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.19.239:9091: connect: connection refused I0423 21:52:11.226125 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 6.880608ms to process 0 records I0423 21:52:11.233363 1 tasks_processing.go:74] worker 24 stopped. I0423 21:52:11.233783 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=a3ae5c1001ba6880f168856cf109408fbb8ad7e6c0b6be72ab96b2ac2ef49800 I0423 21:52:11.233806 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 14.712921ms to process 1 records I0423 21:52:11.249597 1 tasks_processing.go:74] worker 31 stopped. I0423 21:52:11.249621 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 29.25515ms to process 0 records I0423 21:52:11.249632 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 30.93936ms to process 0 records I0423 21:52:11.249639 1 tasks_processing.go:74] worker 0 stopped. I0423 21:52:11.249687 1 tasks_processing.go:74] worker 22 stopped. I0423 21:52:11.249713 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 30.813048ms to process 0 records I0423 21:52:11.249875 1 tasks_processing.go:74] worker 50 stopped. I0423 21:52:11.249894 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 28.81778ms to process 0 records I0423 21:52:11.249903 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 28.847089ms to process 0 records I0423 21:52:11.249909 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 30.418156ms to process 0 records I0423 21:52:11.249915 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 28.036493ms to process 0 records I0423 21:52:11.249920 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 27.61383ms to process 0 records I0423 21:52:11.249922 1 tasks_processing.go:74] worker 51 stopped. I0423 21:52:11.249927 1 tasks_processing.go:74] worker 13 stopped. I0423 21:52:11.249924 1 tasks_processing.go:74] worker 35 stopped. I0423 21:52:11.249943 1 tasks_processing.go:74] worker 38 stopped. I0423 21:52:11.249972 1 tasks_processing.go:74] worker 9 stopped. I0423 21:52:11.250068 1 recorder.go:75] Recording config/image with fingerprint=0dfef521bbe9513859cf19bedcc54fbc491ccbbef70a6d4dab4b1695dc77e55e I0423 21:52:11.250086 1 gather.go:177] gatherer "clusterconfig" function "image" took 31.292577ms to process 1 records E0423 21:52:11.250096 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0423 21:52:11.250103 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 31.348428ms to process 0 records I0423 21:52:11.250109 1 tasks_processing.go:74] worker 12 stopped. I0423 21:52:11.250116 1 tasks_processing.go:74] worker 60 stopped. I0423 21:52:11.250618 1 recorder.go:75] Recording config/infrastructure with fingerprint=6de611b98d3aad284f943371f3308ea78dd75c9933dac6d48bc98153196e25fd I0423 21:52:11.250634 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 29.866418ms to process 1 records I0423 21:52:11.250728 1 tasks_processing.go:74] worker 10 stopped. I0423 21:52:11.250793 1 recorder.go:75] Recording config/oauth with fingerprint=bd1b002295aa32941d0b9d6831ec569c1259a401b201cec3e182948667a61644 I0423 21:52:11.250800 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 31.520361ms to process 1 records I0423 21:52:11.254998 1 tasks_processing.go:74] worker 2 stopped. I0423 21:52:11.255010 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 36.314062ms to process 0 records I0423 21:52:11.255176 1 tasks_processing.go:74] worker 19 stopped. I0423 21:52:11.255462 1 recorder.go:75] Recording config/proxy with fingerprint=87e512b22e41a02f618825881fcf92e188b6caa2baa6ec7a3b9b2e30390f67b6 I0423 21:52:11.255487 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 34.522981ms to process 1 records I0423 21:52:11.255542 1 tasks_processing.go:74] worker 57 stopped. I0423 21:52:11.255585 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 34.122228ms to process 0 records E0423 21:52:11.255662 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0423 21:52:11.255682 1 gather.go:177] gatherer "clusterconfig" function "machines" took 34.035189ms to process 0 records I0423 21:52:11.255694 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 34.076942ms to process 0 records I0423 21:52:11.255712 1 tasks_processing.go:74] worker 17 stopped. I0423 21:52:11.255719 1 tasks_processing.go:74] worker 15 stopped. I0423 21:52:11.255744 1 tasks_processing.go:74] worker 34 stopped. I0423 21:52:11.255758 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 36.936874ms to process 0 records I0423 21:52:11.256034 1 tasks_processing.go:74] worker 54 stopped. I0423 21:52:11.256181 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=e98070bf789dda77f35ee00eebc77dad63639e4ba931fa3d1f2e7a18c417e6fd I0423 21:52:11.256205 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 33.475888ms to process 1 records I0423 21:52:11.256223 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 34.517754ms to process 0 records I0423 21:52:11.256232 1 tasks_processing.go:74] worker 36 stopped. I0423 21:52:11.256355 1 tasks_processing.go:74] worker 25 stopped. I0423 21:52:11.256412 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 34.458791ms to process 0 records I0423 21:52:11.256500 1 tasks_processing.go:74] worker 52 stopped. I0423 21:52:11.256523 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 37.697629ms to process 0 records I0423 21:52:11.257526 1 tasks_processing.go:74] worker 21 stopped. I0423 21:52:11.257540 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 38.202699ms to process 0 records I0423 21:52:11.257618 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=23e1e2e70832574b8ed9aec3666c0131faa46624a0d31cbfc43d25251d4c69f9 I0423 21:52:11.257635 1 tasks_processing.go:74] worker 27 stopped. I0423 21:52:11.257640 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=2e64270d308b791e95515a29522f2971e923bed53f7761f88ab87bac59fc1937 I0423 21:52:11.257661 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 35.934398ms to process 2 records I0423 21:52:11.257766 1 tasks_processing.go:74] worker 47 stopped. I0423 21:52:11.257994 1 recorder.go:75] Recording config/apiserver with fingerprint=9bdccd7a8b259ceb569adc4f92458500a8780539e6746b060c05d8044d2c8024 I0423 21:52:11.258051 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 34.75759ms to process 1 records I0423 21:52:11.258294 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=c8244a26e2cb1d567c34a62247ceaef8964e776044107c96458d8c0d10381168 I0423 21:52:11.258367 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=01c7c0a266db9e26fd5bca2348267b32df4fb89316c2fb35756f9dbce253360a I0423 21:52:11.258467 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=61487509c2b3d5ec0ad6b32788d708839615cb199348ee2b04f65071c480ea81 I0423 21:52:11.258490 1 tasks_processing.go:74] worker 37 stopped. I0423 21:52:11.258495 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=3ddc512753e2537580e43b2615eabc086824a7bf65b110941baedb7b2722b665 I0423 21:52:11.258566 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=0e6ada6d1c1be0fc85d220ceb5442817e30043cb6ace1cfb78b261f42d9af7b6 I0423 21:52:11.258613 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=ecbdb34f01ac56f7c49b7f01ea1f24df63bf5c9cdc65d4c22203eaf804330969 I0423 21:52:11.258657 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=9c167d7d3fd0a7195798a97fc28b49809e7d6741350a6e6f1c45099fda664a0d I0423 21:52:11.258697 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=1ec60d632511a20d8e861f9148ba466fa19f74bb76dee9012fd1e6320fd19ea1 I0423 21:52:11.258733 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=97be20f8b51eda2b1df6399e623a9dd11110e9841623316d835ce9b11fd1ed23 I0423 21:52:11.258783 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=d460834e890c737cf48329eba038dca138314345cda9c23b556d68b3f2b535c7 I0423 21:52:11.258820 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=de98c5d54013f0345f931d2e2be1442b18d446d0c2de4c5547f4c93e3978636a I0423 21:52:11.258844 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=5a5152d369841a4eb636fa489311dbd90ce582a31505a906a1bb1527e3dea292 I0423 21:52:11.258866 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=239a250d5fe819129a0c4f4503e94a41ea02ff5d179f03b3d5f11f5f02498a57 I0423 21:52:11.258875 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 36.271151ms to process 13 records I0423 21:52:11.258979 1 tasks_processing.go:74] worker 41 stopped. E0423 21:52:11.258995 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0423 21:52:11.259053 1 recorder.go:75] Recording config/authentication with fingerprint=1367d9b1eea548b092ff1dc176c90e3d71d2e3198c6bb6ddea5b7a13d0d3a717 I0423 21:52:11.259061 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 38.676594ms to process 1 records I0423 21:52:11.259147 1 tasks_processing.go:74] worker 49 stopped. I0423 21:52:11.259226 1 gather_logs.go:145] no pods in namespace were found I0423 21:52:11.259243 1 recorder.go:75] Recording config/network with fingerprint=2ecc4fbb726b93e1e08e1a60f9cf94129dba66fadbe084e0d62a990a9437ab07 I0423 21:52:11.259254 1 gather.go:177] gatherer "clusterconfig" function "networks" took 36.837385ms to process 1 records I0423 21:52:11.259264 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 37.258635ms to process 0 records E0423 21:52:11.259271 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0423 21:52:11.259275 1 tasks_processing.go:74] worker 14 stopped. I0423 21:52:11.259278 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 36.18129ms to process 0 records I0423 21:52:11.259283 1 tasks_processing.go:74] worker 20 stopped. I0423 21:52:11.259296 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 37.337586ms to process 0 records I0423 21:52:11.259332 1 tasks_processing.go:74] worker 56 stopped. I0423 21:52:11.259412 1 tasks_processing.go:74] worker 55 stopped. I0423 21:52:11.259474 1 recorder.go:75] Recording config/ingress with fingerprint=e34d59496a97c5a47d099c3b4cd95334aa45b035d3e94a8e1be8dcb64b342ac3 I0423 21:52:11.259495 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 37.534557ms to process 1 records I0423 21:52:11.259502 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 36.501109ms to process 0 records I0423 21:52:11.259512 1 tasks_processing.go:74] worker 16 stopped. I0423 21:52:11.259575 1 tasks_processing.go:74] worker 43 stopped. I0423 21:52:11.259588 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=c32ef9ded7e65021d75f64ad136cda1a552e1680634d7ebdef44bd99f50e6d4b I0423 21:52:11.259608 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=54d7d0859e314ebc250038e12d8965c56e391f9b1c09824f4ccb1a19cea0b7be I0423 21:52:11.259625 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=6cc29322828146a66cdce780db064ef55a39320527864c0a0301cb5b53ccd62b I0423 21:52:11.259638 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=125e7492ca4db659e5fb6a46571397200996f8efc8b200a415344f08d6c8cde4 I0423 21:52:11.259663 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=690559c97136016772f2676fe1ed62d2d87eb0fc2958271c1991b9bf2597de12 I0423 21:52:11.259687 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=e837460ac88ed4f6ffd96d873022c6a83a230df8008dff1a499d246c78bd2672 I0423 21:52:11.259711 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=33ebea052c788f2a0877ffabea3389febaa8c7d482f012b5ff91adcd8ebcbb08 I0423 21:52:11.259734 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=2b9e2ba65082b0ee260b53453d246493ced761d6775c898f14c6f9a2158ceb35 I0423 21:52:11.259757 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=c085f8e75a2efe4b37c48ab6fd435671e9692cb8e7bd46d5b464e34d8bad4e83 I0423 21:52:11.259784 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=95822374ec733b9e57c66f7ded61900f65e97548b780a086a6c7d8061ea5e246 I0423 21:52:11.259812 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=1f2a21439f379aecf1dff51be9a31c78a532ae94a33ef927140d4b679d582a83 I0423 21:52:11.259826 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 39.916018ms to process 11 records E0423 21:52:11.259835 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0423 21:52:11.259844 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 38.557236ms to process 0 records I0423 21:52:11.259854 1 tasks_processing.go:74] worker 28 stopped. I0423 21:52:11.260072 1 tasks_processing.go:74] worker 7 stopped. I0423 21:52:11.260088 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 38.620605ms to process 0 records I0423 21:52:11.261401 1 tasks_processing.go:74] worker 4 stopped. I0423 21:52:11.261508 1 recorder.go:75] Recording config/featuregate with fingerprint=91e8977790dbbd00338ec9cf19a1fcc9446479f40e7d62439dd0bb80d94d6062 I0423 21:52:11.261521 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 39.170029ms to process 1 records I0423 21:52:11.261533 1 tasks_processing.go:74] worker 61 stopped. I0423 21:52:11.262537 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=43e4f1b07dbf1996ebcdb4fa6b45048ef2261cdace44090602ae7c24a2f7e156 I0423 21:52:11.262672 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=cec1ec2557c7211432bdea7860a6f431bfd4550cd0174f87815eadf988746d5c I0423 21:52:11.262747 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=c140ce5ae390abc20aaa50b0f9af13fd936a7d2a54bc02f5882b5b003096095d I0423 21:52:11.262778 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 40.064698ms to process 3 records I0423 21:52:11.262819 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 43.103711ms to process 0 records I0423 21:52:11.262847 1 tasks_processing.go:74] worker 26 stopped. I0423 21:52:11.263342 1 tasks_processing.go:74] worker 11 stopped. W0423 21:52:11.263778 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0423 21:52:11.264721 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=621d5125640292f23fdd788d4c88c2dffaa551fc2a5319c91cf7d7d04759db95 I0423 21:52:11.264742 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 43.989049ms to process 1 records I0423 21:52:11.264835 1 tasks_processing.go:74] worker 48 stopped. I0423 21:52:11.265180 1 recorder.go:75] Recording config/node/ip-10-0-0-18.ec2.internal with fingerprint=5414441efb34c262bb9c3101fd79b4a0eb74e362ad4f50c89c9cd02b383ec257 I0423 21:52:11.265273 1 recorder.go:75] Recording config/node/ip-10-0-1-216.ec2.internal with fingerprint=75c34e696b49a6f93dc82222be9bd0bcc9e0bbb38dfa5b5a2d135a0d06412e4c I0423 21:52:11.265290 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0423 21:52:11.265300 1 controller.go:254] Source scaController *sca.Controller is not ready I0423 21:52:11.265305 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0423 21:52:11.265308 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0423 21:52:11.265311 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0423 21:52:11.265326 1 controller.go:531] The operator is still being initialized I0423 21:52:11.265331 1 controller.go:554] The operator is healthy I0423 21:52:11.265355 1 recorder.go:75] Recording config/node/ip-10-0-2-123.ec2.internal with fingerprint=5c8ad715593fc270ef5c8d3a96c57bdf829ba3f4b8e619a51c904dd1777f7cca I0423 21:52:11.265367 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 43.28095ms to process 3 records I0423 21:52:11.265377 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 42.9136ms to process 0 records I0423 21:52:11.265386 1 tasks_processing.go:74] worker 18 stopped. I0423 21:52:11.265743 1 tasks_processing.go:74] worker 58 stopped. I0423 21:52:11.265777 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0423 21:52:11.265792 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 43.38159ms to process 1 records I0423 21:52:11.270039 1 tasks_processing.go:74] worker 40 stopped. E0423 21:52:11.270060 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0423 21:52:11.270103 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0423 21:52:11.270120 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 48.259705ms to process 1 records I0423 21:52:11.270617 1 tasks_processing.go:74] worker 1 stopped. I0423 21:52:11.271900 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=fb42f90ad8572daecc074cef8a32a6256192eb0583fe1ab4798f6fa744ef30e7 I0423 21:52:11.272231 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=e875023c53c020720ffaa2e2b2d222aa84d9a17270c3f0208937f92da173f79b I0423 21:52:11.272250 1 gather.go:177] gatherer "clusterconfig" function "crds" took 51.92715ms to process 2 records I0423 21:52:11.279797 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0423 21:52:11.279810 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0423 21:52:11.279812 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0423 21:52:11.289873 1 base_controller.go:82] Caches are synced for ConfigController I0423 21:52:11.289885 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0423 21:52:11.294819 1 tasks_processing.go:74] worker 45 stopped. I0423 21:52:11.296110 1 recorder.go:75] Recording config/version with fingerprint=6246100da3837dda20bef221a927e30a065016d5ebabac1dd5d7645de53b38ec I0423 21:52:11.296138 1 recorder.go:75] Recording config/id with fingerprint=743928d80056fa06fe23c17627e8f5c4e31ba3b3e4bd49aa262d9419646ac4b9 I0423 21:52:11.296150 1 gather.go:177] gatherer "clusterconfig" function "version" took 73.69443ms to process 2 records I0423 21:52:11.296300 1 tasks_processing.go:74] worker 63 stopped. I0423 21:52:11.296481 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=36885d334cbf7dfd6371d1faec126f2a3c0f88523a9760df20f695a7a0ed8044 I0423 21:52:11.296677 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=8f11b2fb7aa02ea5879f38cd3540b7ac71c426f09b966092436d6f89f5a6a174 I0423 21:52:11.296713 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 74.622789ms to process 2 records I0423 21:52:11.296752 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 77.687256ms to process 0 records I0423 21:52:11.296781 1 tasks_processing.go:74] worker 53 stopped. I0423 21:52:11.297987 1 configmapobserver.go:84] configmaps "insights-config" not found I0423 21:52:11.301409 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0423 21:52:11.304113 1 tasks_processing.go:74] worker 29 stopped. I0423 21:52:11.305308 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-wsk48 with fingerprint=7d54fd7f616627ee761a486fad1336a6d3a7785b89f5c78a00f8684d6f4e7720 I0423 21:52:11.305385 1 recorder.go:75] Recording config/running_containers with fingerprint=7e956a318cb2706ce5a7664396f0adf655eefad4b9d9d9c136ddab727011f3e8 I0423 21:52:11.305395 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 79.888037ms to process 2 records I0423 21:52:11.313233 1 tasks_processing.go:74] worker 3 stopped. I0423 21:52:11.313247 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 92.159185ms to process 0 records I0423 21:52:11.323737 1 tasks_processing.go:74] worker 8 stopped. E0423 21:52:11.323752 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0423 21:52:11.323759 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ps4596k02g87f728elrjhduc7bmtv3r-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ps4596k02g87f728elrjhduc7bmtv3r-primary-cert-bundle-secret" not found I0423 21:52:11.323819 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=968c4cebba8e62a8a522070f64d8d3767d0dfb129da441c5b7779bbae5a33d03 I0423 21:52:11.323831 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 100.751254ms to process 1 records I0423 21:52:11.326022 1 base_controller.go:82] Caches are synced for LoggingSyncer I0423 21:52:11.326037 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0423 21:52:11.328945 1 tasks_processing.go:74] worker 44 stopped. E0423 21:52:11.328962 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0423 21:52:11.328973 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0423 21:52:11.328977 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0423 21:52:11.328988 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0423 21:52:11.329025 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0423 21:52:11.329046 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0423 21:52:11.329051 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0423 21:52:11.329061 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0423 21:52:11.329097 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0423 21:52:11.329104 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0423 21:52:11.329110 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 107.187584ms to process 7 records I0423 21:52:11.331649 1 cluster_transfer.go:102] no available accepted cluster transfer I0423 21:52:11.331664 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0423 21:52:11.346544 1 gather_cluster_operator_pods_and_events.go:121] Found 6 pods with 6 containers I0423 21:52:11.346554 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 4194304 bytes I0423 21:52:11.347059 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-78759f779-9xl9x pod in namespace openshift-ingress (previous: false). I0423 21:52:11.368150 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-789d476894-szb7w pod in namespace openshift-ingress (previous: false). I0423 21:52:11.374493 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0423 21:52:11.381146 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0423 21:52:11.383888 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-789d476894-xmwbj pod in namespace openshift-ingress (previous: false). I0423 21:52:11.433355 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-59cqn pod in namespace openshift-ingress-canary (previous: false). I0423 21:52:11.469552 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-v79dc pod in namespace openshift-ingress-canary (previous: false). I0423 21:52:11.480190 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-w87k2 pod in namespace openshift-ingress-canary (previous: false). I0423 21:52:11.488235 1 sca.go:266] etc-pki-entitlement secret successfully created I0423 21:52:11.488250 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0423 21:52:11.678514 1 tasks_processing.go:74] worker 62 stopped. I0423 21:52:11.678600 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=da36f528667375ba8fe9dc08708f5d7bd68be494e4ceebe88760b9dd262833c0 I0423 21:52:11.678686 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=d9dbb0984bac5d312c2ccedf2dbbb9e226d670b93dec3ac9ee7430d63a068a82 I0423 21:52:11.678727 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=00d1cc3d913b28280b6129f12e8ffce6699aceade0f43d1165f069c130e2f774 I0423 21:52:11.678750 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-78759f779-9xl9x/router_current.log with fingerprint=95bf8a4a1fd3373a681d69cf0ef6e6148cd9597d0902d9534596d8a893822357 I0423 21:52:11.678771 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-789d476894-szb7w/router_current.log with fingerprint=2ae776e1a2529d23154a58bc36c0f82508ba0708d550e04bad9370ff658bed61 I0423 21:52:11.678792 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-789d476894-xmwbj/router_current.log with fingerprint=8c2944195db4b5b17f0363d770f5e47610ee9ff0321a4e8d2ceedd5c9fa44224 I0423 21:52:11.678799 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-59cqn/serve-healthcheck-canary_current.log with fingerprint=9111a2f85d62f976b85e1fced9ed42067b935d48bea0f88fe4ef0c87b6957532 I0423 21:52:11.678803 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-v79dc/serve-healthcheck-canary_current.log with fingerprint=b4336290fd55bcdc6dfd72b205f19dc8c4937e506b44b4e76981b7a2c949c90c I0423 21:52:11.678806 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-w87k2/serve-healthcheck-canary_current.log with fingerprint=248bc6d51f08b2e992176ed1ab09c50eae1b1021495eb5f7bef67c5a68aed929 I0423 21:52:11.678812 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 457.043544ms to process 9 records W0423 21:52:12.261649 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0423 21:52:12.266207 1 tasks_processing.go:74] worker 46 stopped. I0423 21:52:12.266229 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0423 21:52:12.266243 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.046826941s to process 1 records I0423 21:52:12.266278 1 recorder.go:75] Recording config/alerts with fingerprint=aec1c20f8858bee218a6bc71c79a98663cd122e7aed41beec59b78ffe9de75d7 I0423 21:52:12.266285 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.046799755s to process 1 records I0423 21:52:12.266291 1 tasks_processing.go:74] worker 32 stopped. W0423 21:52:13.261867 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0423 21:52:13.708429 1 tasks_processing.go:74] worker 59 stopped. I0423 21:52:13.708547 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=3d4399df059c6060a982fdfc42e2ba38c232b2ac31c1a6e39736a9a4155abdd1 I0423 21:52:13.708595 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0423 21:52:13.708629 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=6cc64dfae173b7733081a0efd326431f8c7a2f0325bb5812c5ec0bb54dfb12e0 I0423 21:52:13.708640 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0423 21:52:13.708661 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=a1957c166ef461a2afe3d27c38bbe608f6d0a2ab29e07510d8bb382c11f49528 I0423 21:52:13.708677 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0423 21:52:13.708696 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=f6d4140c6423119b7b965e2fa91c6f4b7ac875ee03a5ff8ee5e926e4ebc96329 I0423 21:52:13.708722 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=e4d0fee66222a1521c0ab6ebee72255e4e3aa329c81239025c841d604416c1ec I0423 21:52:13.708746 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=3808d4c807caa0c54ebc2d6d0c12efb0aeb3d1c636972ecf7c688b7fc1642929 I0423 21:52:13.708775 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=2ab909465a4e22428581434c19baa544c25a7bcba337093c8a94d3fc3bf59085 I0423 21:52:13.709031 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=2e3bd086710c3f396845055954b44ec37e00cd3dde89d3ab6ce3e83d41bad8b5 I0423 21:52:13.709049 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=f118e9b798c5cfab7d8abd82871e572babfa0efffee20395abe4be4903685534 I0423 21:52:13.709056 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0423 21:52:13.709073 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=6c928b52c6472d956c73a71b1ce0cf16ec291bffa34750d580c08de9a563ee93 I0423 21:52:13.709083 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0423 21:52:13.709100 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=6564ba20967cdb29034ef356c7131a15f586fcf50b70c2bd87fad64360a511b2 I0423 21:52:13.709108 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0423 21:52:13.709127 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=cb0b0f8df285d180a73ff905b3026df437d4e414dfb2d3f996c43f659211bf90 I0423 21:52:13.709137 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0423 21:52:13.709171 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=69d7c6db99a6dfd33778aab7cff659afe19a61cdec560e091c9ede4e0eaed0e0 I0423 21:52:13.709274 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=279b6ceb3bd094e4a07217c598afa0c660939d9200b7cf8e30db6fd69b25b946 I0423 21:52:13.709284 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0423 21:52:13.709290 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0423 21:52:13.709311 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0423 21:52:13.709336 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=0e3a5ba7f1c6db6e452f65467945b94f98522c6393251a86c8e9d74f6bf10a62 I0423 21:52:13.709358 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=be30add76a2c8d4070f8f3b2c9431cfd51d264452b3b8da5d46d95c8861233c1 I0423 21:52:13.709368 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0423 21:52:13.709384 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=9e5da0f906a316a0c0b92b7d6085ed2c0f8c4f715cc962bd59d32710cedea36c I0423 21:52:13.709393 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0423 21:52:13.709411 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=e7770665c3c543b4f0c9e542455d18c3a7ed6c94371ceb04aa3572ac1e04df5a I0423 21:52:13.709421 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0423 21:52:13.709433 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=5b053b205981bca204a3eaafb09ef838bdecd3012cb13c9afb671c2a64103d17 I0423 21:52:13.709446 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=fa01cc8cb2dd87506e275b0c296aba867738f04cc70dd4e565e77bedbedd42d0 I0423 21:52:13.709461 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=99903ab7b3b27900c33307623819a848352e5f7335b1b763c8f88730d53d08cd I0423 21:52:13.709480 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=bffed19374849391f4e93b2892d5899d544c83e1ecc7377f6e4cc14329cb0e47 I0423 21:52:13.709490 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0423 21:52:13.709514 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=bf6c0e78359516e92187a0e59e5aa3cd3efdb65ec0c3bf5309ddd16c9b6de1b2 I0423 21:52:13.709530 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0423 21:52:13.709537 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0423 21:52:13.709544 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.487181914s to process 39 records W0423 21:52:14.262182 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0423 21:52:15.262106 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0423 21:52:16.262308 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0423 21:52:16.262346 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0423 21:52:16.262362 1 tasks_processing.go:74] worker 30 stopped. E0423 21:52:16.262371 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0423 21:52:16.262382 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0423 21:52:16.262425 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.039857791s to process 1 records I0423 21:52:24.077314 1 tasks_processing.go:74] worker 39 stopped. I0423 21:52:24.077353 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0423 21:52:24.077365 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.854660932s to process 1 records I0423 21:52:24.827392 1 tasks_processing.go:74] worker 5 stopped. I0423 21:52:24.827660 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=80e5b054934e37ddf232a6b52185c88ef3149965af63f70b0125fd661e8f707e I0423 21:52:24.827676 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.608775441s to process 1 records E0423 21:52:24.827741 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.609s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0423 21:52:24.828847 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0423 21:52:24.828861 1 periodic.go:216] Running workloads gatherer I0423 21:52:24.828876 1 tasks_processing.go:45] number of workers: 2 I0423 21:52:24.828884 1 tasks_processing.go:69] worker 1 listening for tasks. I0423 21:52:24.828888 1 tasks_processing.go:71] worker 1 working on workload_info task. I0423 21:52:24.828900 1 tasks_processing.go:69] worker 0 listening for tasks. I0423 21:52:24.828921 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0423 21:52:24.835128 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-18.ec2.internal... I0423 21:52:24.835135 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-216.ec2.internal... I0423 21:52:24.835145 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-123.ec2.internal... I0423 21:52:24.855279 1 tasks_processing.go:74] worker 0 stopped. I0423 21:52:24.855293 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.33911ms to process 0 records I0423 21:52:29.806358 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0423 21:52:29.834254 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0423 21:52:29.842723 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (10ms) I0423 21:52:29.851211 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (8ms) I0423 21:52:29.859281 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (8ms) I0423 21:52:29.867355 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (8ms) I0423 21:52:29.875400 1 gather_workloads_info.go:387] No image sha256:1f118a4ec0b92cb88c8544ea31d2b9adfc7006eeddb7a8d260150ecbce4adb8c (8ms) I0423 21:52:29.883337 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (8ms) I0423 21:52:29.891513 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (8ms) I0423 21:52:29.899883 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (8ms) I0423 21:52:29.907790 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (8ms) I0423 21:52:29.915594 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (8ms) I0423 21:52:29.943549 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (28ms) I0423 21:52:30.042868 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (99ms) I0423 21:52:30.143222 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (100ms) I0423 21:52:30.241885 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (99ms) I0423 21:52:30.342473 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (101ms) I0423 21:52:30.442404 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (100ms) I0423 21:52:30.542782 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (100ms) I0423 21:52:30.642801 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (100ms) I0423 21:52:30.742105 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (99ms) I0423 21:52:30.842373 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (100ms) I0423 21:52:30.942415 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (100ms) I0423 21:52:31.043419 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (101ms) I0423 21:52:31.142835 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (99ms) I0423 21:52:31.242708 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (100ms) I0423 21:52:31.343455 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (101ms) I0423 21:52:31.442894 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (99ms) I0423 21:52:31.542207 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (99ms) I0423 21:52:31.642338 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (100ms) I0423 21:52:31.742690 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (100ms) I0423 21:52:31.846948 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (104ms) I0423 21:52:31.942676 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (96ms) I0423 21:52:32.042583 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (100ms) I0423 21:52:32.143525 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (101ms) I0423 21:52:32.242371 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0423 21:52:32.342364 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0423 21:52:32.442183 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (100ms) I0423 21:52:32.542674 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (100ms) I0423 21:52:32.642886 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0423 21:52:32.742239 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (99ms) I0423 21:52:32.842787 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (101ms) I0423 21:52:32.943155 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (100ms) I0423 21:52:33.042743 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (100ms) I0423 21:52:33.142426 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (100ms) I0423 21:52:33.241788 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (99ms) I0423 21:52:33.342676 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (101ms) I0423 21:52:33.442543 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (100ms) I0423 21:52:33.542813 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (100ms) I0423 21:52:33.542845 1 tasks_processing.go:74] worker 1 stopped. E0423 21:52:33.542855 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.128.0.10:8443/gather_runtime_info I0423 21:52:33.543130 1 recorder.go:75] Recording config/workload_info with fingerprint=d3c94a5ba5c71ac02e1319facaed112964e95c62d8247fbec1f33197955d8f80 I0423 21:52:33.543145 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.713950788s to process 1 records E0423 21:52:33.543187 1 periodic.go:254] "Unhandled Error" err="workloads failed after 8.714s with: function \"workload_info\" failed with an error" I0423 21:52:33.544289 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0423 21:52:33.544302 1 periodic.go:216] Running conditional gatherer I0423 21:52:33.550576 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0423 21:52:33.780609 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0423 21:52:34.858707 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0423 21:52:34.864298 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0423 21:52:34.864325 1 tasks_processing.go:45] number of workers: 3 I0423 21:52:34.864335 1 tasks_processing.go:69] worker 2 listening for tasks. I0423 21:52:34.864339 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0423 21:52:34.864345 1 tasks_processing.go:69] worker 0 listening for tasks. I0423 21:52:34.864357 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0423 21:52:34.864356 1 tasks_processing.go:69] worker 1 listening for tasks. I0423 21:52:34.864363 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0423 21:52:34.864367 1 tasks_processing.go:74] worker 1 stopped. I0423 21:52:34.864427 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0423 21:52:34.864442 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 924ns to process 1 records I0423 21:52:34.864476 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0423 21:52:34.864484 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.475µs to process 1 records I0423 21:52:34.864489 1 tasks_processing.go:74] worker 0 stopped. I0423 21:52:34.864616 1 tasks_processing.go:74] worker 2 stopped. I0423 21:52:34.864624 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 243.287µs to process 0 records I0423 21:52:34.864648 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0423 21:52:34.864664 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0423 21:52:34.894838 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=2714563a2e3e72e08b88c1c5336c61e15a119cd4ac91aa39af1fc6c72a3f27a3 I0423 21:52:34.894985 1 diskrecorder.go:70] Writing 119 records to /var/lib/insights-operator/insights-2026-04-23-215234.tar.gz I0423 21:52:34.902462 1 diskrecorder.go:51] Wrote 119 records to disk in 7ms I0423 21:52:34.902493 1 periodic.go:285] Gathering cluster info every 2h0m0s I0423 21:52:34.902508 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0423 21:53:41.218771 1 diskrecorder.go:223] Found files to send: insights-2026-04-23-215234.tar.gz I0423 21:53:41.218801 1 insightsuploader.go:150] Checking archives to upload periodically every 16m16.117511232s I0423 21:53:41.218809 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0423 21:53:41.229402 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0423 21:53:41.664465 1 requests.go:87] Successfully reported id=2026-04-23T21:53:41Z x-rh-insights-request-id=eb3c3f18ae434670a75ed3b14163e4ae, wrote=53889 I0423 21:53:41.664578 1 insightsuploader.go:187] Uploaded report successfully in 445.758621ms I0423 21:53:41.664602 1 controller.go:129] Initializing last reported time to 2026-04-23T21:53:41Z I0423 21:53:41.664671 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0423 21:53:41.664680 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0423 21:53:41.664687 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0423 21:53:41.671063 1 controller.go:554] The operator is healthy I0423 21:54:11.233793 1 controller.go:554] The operator is healthy I0423 21:54:45.269618 1 insightsreport.go:137] Pulling report from smart-proxy I0423 21:54:45.269643 1 insightsreport.go:149] Retrieving report I0423 21:54:45.276257 1 requests.go:110] Retrieving report for cluster: 1b720102-e5de-42cf-8f05-7473212e02ec I0423 21:54:45.276270 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/1b720102-e5de-42cf-8f05-7473212e02ec/reports I0423 21:54:45.279326 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/1b720102-e5de-42cf-8f05-7473212e02ec/reports I0423 21:54:45.491388 1 insightsreport.go:184] Report retrieved I0423 21:54:45.518452 1 insightsreport.go:239] Report retrieved correctly