W0319 18:03:15.582355 1 cmd.go:257] Using insecure, self-signed certificates I0319 18:03:16.282231 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:16.282664 1 observer_polling.go:159] Starting file observer I0319 18:03:17.360686 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0319 18:03:17.361018 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0319 18:03:17.362746 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0319 18:03:17.363285 1 secure_serving.go:57] Forcing use of http/1.1 only W0319 18:03:17.363573 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0319 18:03:17.363585 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0319 18:03:17.363591 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0319 18:03:17.363595 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0319 18:03:17.363599 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0319 18:03:17.363603 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0319 18:03:17.367228 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0319 18:03:17.367246 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"7c9166ba-cfaa-401e-92f5-fad6de28c9ab", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0319 18:03:17.368845 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0319 18:03:17.368865 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0319 18:03:17.368896 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0319 18:03:17.368940 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 18:03:17.369221 1 secure_serving.go:213] Serving securely on [::]:8443 I0319 18:03:17.369231 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3992060234/tls.crt::/tmp/serving-cert-3992060234/tls.key" I0319 18:03:17.369262 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0319 18:03:17.369294 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0319 18:03:17.369514 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file W0319 18:03:17.372260 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0319 18:03:17.372306 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 18:03:17.372459 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0319 18:03:17.377416 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 18:03:17.377461 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 18:03:17.381856 1 secretconfigobserver.go:119] support secret does not exist I0319 18:03:17.387109 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 18:03:17.391150 1 secretconfigobserver.go:119] support secret does not exist I0319 18:03:17.394346 1 recorder.go:176] Pruning old reports every 4h24m30s, max age is 288h0m0s I0319 18:03:17.399570 1 periodic.go:216] Running clusterconfig gatherer I0319 18:03:17.399601 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0319 18:03:17.399624 1 tasks_processing.go:45] number of workers: 32 I0319 18:03:17.399618 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0319 18:03:17.399649 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0319 18:03:17.399654 1 tasks_processing.go:69] worker 31 listening for tasks. I0319 18:03:17.399658 1 insightsreport.go:296] Starting report retriever I0319 18:03:17.399663 1 tasks_processing.go:69] worker 16 listening for tasks. I0319 18:03:17.399667 1 tasks_processing.go:69] worker 15 listening for tasks. I0319 18:03:17.399658 1 tasks_processing.go:71] worker 31 working on proxies task. I0319 18:03:17.399672 1 tasks_processing.go:69] worker 17 listening for tasks. I0319 18:03:17.399675 1 tasks_processing.go:71] worker 15 working on storage_cluster task. I0319 18:03:17.399679 1 tasks_processing.go:69] worker 18 listening for tasks. I0319 18:03:17.399684 1 tasks_processing.go:69] worker 19 listening for tasks. I0319 18:03:17.399689 1 tasks_processing.go:69] worker 20 listening for tasks. I0319 18:03:17.399694 1 tasks_processing.go:69] worker 21 listening for tasks. I0319 18:03:17.399699 1 tasks_processing.go:69] worker 22 listening for tasks. I0319 18:03:17.399705 1 tasks_processing.go:69] worker 23 listening for tasks. I0319 18:03:17.399713 1 tasks_processing.go:69] worker 24 listening for tasks. I0319 18:03:17.399718 1 tasks_processing.go:69] worker 25 listening for tasks. I0319 18:03:17.399723 1 tasks_processing.go:69] worker 26 listening for tasks. I0319 18:03:17.399728 1 tasks_processing.go:69] worker 27 listening for tasks. I0319 18:03:17.399732 1 tasks_processing.go:69] worker 28 listening for tasks. I0319 18:03:17.399737 1 tasks_processing.go:69] worker 29 listening for tasks. I0319 18:03:17.399744 1 tasks_processing.go:69] worker 30 listening for tasks. I0319 18:03:17.399752 1 tasks_processing.go:69] worker 7 listening for tasks. I0319 18:03:17.399756 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 18:03:17.399761 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 18:03:17.399768 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 18:03:17.399775 1 tasks_processing.go:69] worker 3 listening for tasks. I0319 18:03:17.399779 1 tasks_processing.go:69] worker 4 listening for tasks. I0319 18:03:17.399784 1 tasks_processing.go:69] worker 5 listening for tasks. I0319 18:03:17.399788 1 tasks_processing.go:69] worker 6 listening for tasks. I0319 18:03:17.399794 1 tasks_processing.go:69] worker 11 listening for tasks. I0319 18:03:17.399798 1 tasks_processing.go:69] worker 8 listening for tasks. I0319 18:03:17.399803 1 tasks_processing.go:69] worker 9 listening for tasks. I0319 18:03:17.399807 1 tasks_processing.go:69] worker 10 listening for tasks. I0319 18:03:17.399812 1 tasks_processing.go:69] worker 13 listening for tasks. I0319 18:03:17.399817 1 tasks_processing.go:69] worker 12 listening for tasks. I0319 18:03:17.399821 1 tasks_processing.go:69] worker 14 listening for tasks. I0319 18:03:17.399668 1 tasks_processing.go:71] worker 16 working on operators task. I0319 18:03:17.399953 1 tasks_processing.go:71] worker 14 working on openstack_version task. I0319 18:03:17.399665 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0319 18:03:17.400042 1 tasks_processing.go:71] worker 30 working on ingress task. I0319 18:03:17.400124 1 tasks_processing.go:71] worker 7 working on ingress_certificates task. I0319 18:03:17.400166 1 tasks_processing.go:71] worker 6 working on machine_config_pools task. I0319 18:03:17.400262 1 tasks_processing.go:71] worker 17 working on jaegers task. I0319 18:03:17.400398 1 tasks_processing.go:71] worker 0 working on schedulers task. I0319 18:03:17.400726 1 tasks_processing.go:71] worker 3 working on clusterroles task. I0319 18:03:17.400823 1 tasks_processing.go:71] worker 4 working on authentication task. I0319 18:03:17.400971 1 tasks_processing.go:71] worker 2 working on validating_webhook_configurations task. I0319 18:03:17.401093 1 tasks_processing.go:71] worker 10 working on aggregated_monitoring_cr_names task. I0319 18:03:17.401431 1 tasks_processing.go:71] worker 13 working on machine_configs task. I0319 18:03:17.401501 1 tasks_processing.go:71] worker 9 working on sap_pods task. I0319 18:03:17.401575 1 tasks_processing.go:71] worker 5 working on tsdb_status task. I0319 18:03:17.401588 1 tasks_processing.go:71] worker 11 working on container_images task. W0319 18:03:17.401607 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:17.401624 1 tasks_processing.go:71] worker 5 working on openshift_logging task. I0319 18:03:17.401337 1 tasks_processing.go:71] worker 12 working on config_maps task. I0319 18:03:17.402566 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 32.819µs to process 0 records I0319 18:03:17.402680 1 tasks_processing.go:71] worker 23 working on openstack_dataplanedeployments task. I0319 18:03:17.401511 1 tasks_processing.go:71] worker 8 working on openstack_dataplanenodesets task. I0319 18:03:17.403073 1 tasks_processing.go:71] worker 21 working on machines task. I0319 18:03:17.403130 1 tasks_processing.go:71] worker 22 working on openstack_controlplanes task. I0319 18:03:17.403286 1 tasks_processing.go:71] worker 26 working on olm_operators task. I0319 18:03:17.403079 1 tasks_processing.go:71] worker 20 working on dvo_metrics task. I0319 18:03:17.403456 1 tasks_processing.go:71] worker 28 working on image task. I0319 18:03:17.403541 1 tasks_processing.go:71] worker 29 working on infrastructures task. I0319 18:03:17.403085 1 tasks_processing.go:71] worker 18 working on machine_sets task. I0319 18:03:17.403096 1 tasks_processing.go:71] worker 19 working on nodenetworkstates task. I0319 18:03:17.403733 1 tasks_processing.go:71] worker 24 working on service_accounts task. I0319 18:03:17.403829 1 tasks_processing.go:71] worker 27 working on certificate_signing_requests task. I0319 18:03:17.403955 1 tasks_processing.go:71] worker 25 working on pod_network_connectivity_checks task. I0319 18:03:17.404186 1 tasks_processing.go:71] worker 1 working on install_plans task. I0319 18:03:17.404413 1 tasks_processing.go:71] worker 6 working on sap_config task. I0319 18:03:17.404437 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 4.232348ms to process 0 records I0319 18:03:17.405763 1 tasks_processing.go:71] worker 14 working on cluster_apiserver task. I0319 18:03:17.405821 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 5.798315ms to process 0 records I0319 18:03:17.405839 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 6.135411ms to process 0 records I0319 18:03:17.405869 1 tasks_processing.go:71] worker 15 working on overlapping_namespace_uids task. I0319 18:03:17.407270 1 tasks_processing.go:71] worker 17 working on image_registries task. I0319 18:03:17.407594 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0319 18:03:17.407618 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 18:03:17.407624 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0319 18:03:17.407628 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0319 18:03:17.407646 1 controller.go:531] The operator is still being initialized I0319 18:03:17.407649 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 6.996486ms to process 0 records I0319 18:03:17.407657 1 controller.go:554] The operator is healthy I0319 18:03:17.407880 1 tasks_processing.go:71] worker 30 working on node_logs task. I0319 18:03:17.408005 1 recorder.go:75] Recording config/ingress with fingerprint=202cd7ba7ce2b1b13fdc8f97a098353f29c9d2e5cb0a8627855ad1b27775a390 I0319 18:03:17.408024 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 7.378753ms to process 1 records I0319 18:03:17.408095 1 recorder.go:75] Recording config/proxy with fingerprint=6eb140ae5d2c8d44bb0848be867b574d048e0c3eedf35594146df58d63597b56 I0319 18:03:17.408103 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 8.206514ms to process 1 records I0319 18:03:17.408112 1 tasks_processing.go:71] worker 31 working on nodenetworkconfigurationpolicies task. I0319 18:03:17.412633 1 tasks_processing.go:71] worker 23 working on active_alerts task. W0319 18:03:17.412678 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:17.412713 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 9.921231ms to process 0 records I0319 18:03:17.412722 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 10.597453ms to process 0 records I0319 18:03:17.412728 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 37.519µs to process 0 records I0319 18:03:17.412737 1 tasks_processing.go:71] worker 23 working on oauths task. I0319 18:03:17.412930 1 tasks_processing.go:71] worker 5 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0319 18:03:17.412974 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 9.899047ms to process 0 records I0319 18:03:17.412965 1 tasks_processing.go:71] worker 8 working on networks task. I0319 18:03:17.413516 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=b014422f16d1821d1c54d311f6464a83ce4d5016dc2807049b40c5e2c05d7c7a I0319 18:03:17.413551 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 13.013591ms to process 1 records I0319 18:03:17.413423 1 tasks_processing.go:71] worker 0 working on lokistack task. I0319 18:03:17.413573 1 tasks_processing.go:71] worker 18 working on feature_gates task. I0319 18:03:17.413567 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 9.868845ms to process 0 records I0319 18:03:17.414263 1 tasks_processing.go:71] worker 4 working on machine_healthchecks task. I0319 18:03:17.414798 1 recorder.go:75] Recording config/authentication with fingerprint=96edb56143f43f95336fa2a8c53d0a77d40dac39d2faba9bb924f7b6fad31ec9 I0319 18:03:17.414850 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 13.379501ms to process 1 records I0319 18:03:17.415078 1 tasks_processing.go:71] worker 31 working on metrics task. I0319 18:03:17.415119 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 6.957396ms to process 0 records W0319 18:03:17.415167 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:17.415201 1 tasks_processing.go:71] worker 31 working on mutating_webhook_configurations task. I0319 18:03:17.415220 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 70.529µs to process 0 records I0319 18:03:17.415652 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 11.202157ms to process 0 records I0319 18:03:17.415630 1 tasks_processing.go:71] worker 6 working on qemu_kubevirt_launcher_logs task. I0319 18:03:17.415869 1 tasks_processing.go:71] worker 22 working on cost_management_metrics_configs task. I0319 18:03:17.415935 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 12.719541ms to process 0 records E0319 18:03:17.416207 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0319 18:03:17.416254 1 gather.go:177] gatherer "clusterconfig" function "machines" took 12.939758ms to process 0 records I0319 18:03:17.416229 1 tasks_processing.go:71] worker 21 working on image_pruners task. I0319 18:03:17.416402 1 tasks_processing.go:71] worker 9 working on sap_datahubs task. I0319 18:03:17.416532 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 14.855225ms to process 0 records I0319 18:03:17.416589 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 12.86645ms to process 0 records I0319 18:03:17.416609 1 tasks_processing.go:71] worker 19 working on support_secret task. E0319 18:03:17.416616 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0319 18:03:17.416788 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 12.60481ms to process 0 records I0319 18:03:17.416623 1 tasks_processing.go:71] worker 25 working on nodes task. I0319 18:03:17.418777 1 tasks_processing.go:71] worker 29 working on ceph_cluster task. I0319 18:03:17.419558 1 recorder.go:75] Recording config/infrastructure with fingerprint=97f8ad4129efd47afa6fff22045d3257e74ae02e8d9ea1e86fc16ed9aec0c724 I0319 18:03:17.419578 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 15.215164ms to process 1 records I0319 18:03:17.422836 1 tasks_processing.go:71] worker 13 working on monitoring_persistent_volumes task. E0319 18:03:17.422913 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0319 18:03:17.422942 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0319 18:03:17.422953 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 21.382208ms to process 1 records I0319 18:03:17.423134 1 tasks_processing.go:71] worker 2 working on silenced_alerts task. W0319 18:03:17.423167 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:17.423385 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=4d1f9db6b932fcdd6be740b44774ee9ba7c32adce807afbfd3416c693125d93a I0319 18:03:17.423516 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=241872ad310f953546f36f578ef034502a7e446d58ccb2c7fec7b5c453699517 I0319 18:03:17.423585 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=9fb4dc1e22ecb5d89aa9b1e13dd48ec50320916e62519772740d6489c4a529c7 I0319 18:03:17.423647 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=fb25a5b5727c7c092982bc40b87651808d84861ec358a9ee753d948d9a573d5d I0319 18:03:17.423702 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=62dce52a92343bcabc6e010a5ec7a00d80160718ec6cfe9f14b52005bf751bb8 I0319 18:03:17.423776 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=d74799f8e5bfda85a01969aa03728bb9671a0d728f3d89847ada796ce38621ea I0319 18:03:17.423838 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=5616670a2103ec0c029a23772208115d32f77498465dbce1238698e074ba4e10 I0319 18:03:17.423925 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=867440b396b834768f5eac8a800d9826c905142fa7f28d5f6d7a192eeba6ee98 I0319 18:03:17.423994 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=112e94386fb639cd3073829741d77c10e6880d80828660853df1edb00dc8142a I0319 18:03:17.424052 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=4dd09c0a5d9f403dec98efe225186721f0e910caf782a50c0a730b0f7e2cd81c I0319 18:03:17.424105 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=4c8887f6294b0d5f3c2757cb25346ac977ddf796ec2c4488a8255a6bf0a231b4 I0319 18:03:17.424134 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 22.096054ms to process 11 records I0319 18:03:17.424162 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 29.843µs to process 0 records I0319 18:03:17.424189 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 10.360625ms to process 0 records I0319 18:03:17.424194 1 tasks_processing.go:71] worker 2 working on crds task. I0319 18:03:17.424311 1 tasks_processing.go:71] worker 28 working on version task. I0319 18:03:17.424203 1 tasks_processing.go:71] worker 0 working on container_runtime_configs task. I0319 18:03:17.424305 1 recorder.go:75] Recording config/image with fingerprint=d8eb33b6118dddb054a4530d7048f3fe55f49c17e7b54ea86702b471fd5d73cb I0319 18:03:17.425187 1 gather.go:177] gatherer "clusterconfig" function "image" took 20.719071ms to process 1 records E0319 18:03:17.425219 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0319 18:03:17.425262 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 9.955175ms to process 0 records I0319 18:03:17.425292 1 tasks_processing.go:71] worker 4 working on pdbs task. I0319 18:03:17.425796 1 tasks_processing.go:71] worker 22 working on storage_classes task. I0319 18:03:17.426419 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.830381ms to process 0 records I0319 18:03:17.426479 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 9.814235ms to process 0 records I0319 18:03:17.426607 1 tasks_processing.go:71] worker 9 working on machine_autoscalers task. I0319 18:03:17.433401 1 tasks_processing.go:71] worker 14 working on node_features task. I0319 18:03:17.433829 1 recorder.go:75] Recording config/apiserver with fingerprint=db9689824aef6c136c4b6eb19793da6b798c638574c32c1cdb41875bec3269d2 I0319 18:03:17.433884 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 27.618425ms to process 1 records I0319 18:03:17.434111 1 tasks_processing.go:71] worker 17 working on openshift_machine_api_events task. I0319 18:03:17.434630 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=58448b464ac7d33d6846b219f6f8081c214ed05a376f3491322c64f879a94158 I0319 18:03:17.434678 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 26.555099ms to process 1 records I0319 18:03:17.434779 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c2ee9b82f56ef151198c7c525268c0077d232328a64534614d6d681254b83756 I0319 18:03:17.434798 1 tasks_processing.go:71] worker 21 working on operators_pods_and_events task. I0319 18:03:17.434817 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 17.670926ms to process 1 records I0319 18:03:17.434989 1 tasks_processing.go:74] worker 8 stopped. I0319 18:03:17.435154 1 recorder.go:75] Recording config/network with fingerprint=68d3501e1953f09c72e2bc08d1a804480363ddaa87540409d90e59f6b1cf7065 I0319 18:03:17.435193 1 gather.go:177] gatherer "clusterconfig" function "networks" took 20.701768ms to process 1 records I0319 18:03:17.435307 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=25f1daff4113dba9826f43e6711baaf2f00778753e867ce6c2adb17f3012eae2 I0319 18:03:17.435357 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=336925f2af58d9b7367bfed6c1599b7662364d282c5ad3a096306c3385454995 I0319 18:03:17.435400 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e7e26f4ce5d9753f3607288322714adf522977148c740a852b55695b50da4848 I0319 18:03:17.435429 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 9.180666ms to process 3 records I0319 18:03:17.435455 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 10.004773ms to process 0 records I0319 18:03:17.435471 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0319 18:03:17.435323 1 tasks_processing.go:74] worker 4 stopped. E0319 18:03:17.435480 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0319 18:03:17.435686 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 18.099377ms to process 0 records I0319 18:03:17.435716 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 26.900151ms to process 0 records I0319 18:03:17.435741 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 31.63358ms to process 0 records I0319 18:03:17.435766 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 33.886289ms to process 0 records I0319 18:03:17.435791 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.305297ms to process 0 records I0319 18:03:17.435814 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 12.305286ms to process 0 records I0319 18:03:17.435849 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 8.518722ms to process 0 records I0319 18:03:17.436184 1 recorder.go:75] Recording config/oauth with fingerprint=a83528022b2643f1db2458a7f41cd8278cb8da80a3d6f979f865365969884c48 I0319 18:03:17.436228 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 22.834817ms to process 1 records I0319 18:03:17.436258 1 tasks_processing.go:74] worker 23 stopped. I0319 18:03:17.435486 1 tasks_processing.go:74] worker 19 stopped. I0319 18:03:17.435491 1 tasks_processing.go:74] worker 0 stopped. I0319 18:03:17.435511 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0319 18:03:17.436449 1 tasks_processing.go:74] worker 30 stopped. I0319 18:03:17.436478 1 tasks_processing.go:74] worker 26 stopped. I0319 18:03:17.436502 1 tasks_processing.go:74] worker 10 stopped. I0319 18:03:17.436525 1 tasks_processing.go:74] worker 29 stopped. W0319 18:03:17.435823 1 operator.go:328] started I0319 18:03:17.435854 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0319 18:03:17.436548 1 tasks_processing.go:74] worker 9 stopped. I0319 18:03:17.436553 1 tasks_processing.go:74] worker 13 stopped. I0319 18:03:17.436698 1 tasks_processing.go:74] worker 18 stopped. I0319 18:03:17.436842 1 recorder.go:75] Recording config/featuregate with fingerprint=9998c72139e137c492fea284e0aa85cc17ab3d8357b1eee57732f0952df45826 I0319 18:03:17.436863 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 23.115301ms to process 1 records I0319 18:03:17.437013 1 tasks_processing.go:74] worker 25 stopped. E0319 18:03:17.437255 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0319 18:03:17.437422 1 recorder.go:75] Recording config/node/ip-10-0-0-199.ec2.internal with fingerprint=f88725baeaf8468b2306c0737bab49c8b8a1e584188ef6d9da2585f2b5e026fd I0319 18:03:17.437553 1 recorder.go:75] Recording config/node/ip-10-0-1-10.ec2.internal with fingerprint=3c75a6ff8414929b3ef6db9ce9d701bf613bba367da7ebc1908852f6dda052b6 I0319 18:03:17.437697 1 recorder.go:75] Recording config/node/ip-10-0-2-8.ec2.internal with fingerprint=62c629f0bb586c8630e14bc274dabb25564c79470e99086c57fdb4b6cee8fc6a I0319 18:03:17.437729 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 20.155607ms to process 3 records I0319 18:03:17.437755 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 3.853093ms to process 0 records I0319 18:03:17.437868 1 tasks_processing.go:74] worker 14 stopped. I0319 18:03:17.437977 1 tasks_processing.go:74] worker 3 stopped. I0319 18:03:17.438030 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=0ed5b88d56b07ed1bdd167e461e7ced4a26358fca3cd93afa988fdadcbc5d55c I0319 18:03:17.439080 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=3887d9285b2ceff0f8e59f60fd508094f258bee1a83b46aad2a216273bfdba01 I0319 18:03:17.439135 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 36.71374ms to process 2 records I0319 18:03:17.439490 1 tasks_processing.go:74] worker 27 stopped. I0319 18:03:17.439516 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 35.634444ms to process 0 records W0319 18:03:17.441884 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 18:03:17.444141 1 tasks_processing.go:74] worker 17 stopped. I0319 18:03:17.444196 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 9.970983ms to process 0 records I0319 18:03:17.444268 1 tasks_processing.go:74] worker 22 stopped. I0319 18:03:17.444390 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=01306b8aa1d3bb8516f5d597feba293a82eb9a9ed0878c34cf3e94f474f46f7d I0319 18:03:17.444429 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=5233f5cd84261755c9cd79755916169ae4de754d45ea4b1de52a35abf3ea163b I0319 18:03:17.444457 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 18.424713ms to process 2 records I0319 18:03:17.444613 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=670a02ed77b1e47b13d75ad19a9140bb4654ae7415bd7dceee9e50816effe427 I0319 18:03:17.444886 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ad0f50fa17a6989bc9f8f7c283c66e596058e02c81a39d57465db6acf8f09578 I0319 18:03:17.444991 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=4113b9b2bd117018074ee738b0f1d4369f00b08a29e69448df666b867d835bdc I0319 18:03:17.445022 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 29.04333ms to process 3 records I0319 18:03:17.445222 1 tasks_processing.go:74] worker 31 stopped. I0319 18:03:17.446100 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0319 18:03:17.446156 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 18:03:17.446184 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0319 18:03:17.446214 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0319 18:03:17.446239 1 controller.go:254] Source scaController *sca.Controller is not ready I0319 18:03:17.446304 1 controller.go:531] The operator is still being initialized I0319 18:03:17.446333 1 controller.go:554] The operator is healthy I0319 18:03:17.447598 1 tasks_processing.go:74] worker 2 stopped. I0319 18:03:17.449112 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=4d9f616bcf7e14e7cd3aa4c57a9cd3b9dedee830ddd321ce0638e9ef632ee571 I0319 18:03:17.449819 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=a0586fb9fa4338762094a60160ed5f6551c9cdd4d8f296812cf6dc7dd8cd12d7 I0319 18:03:17.449871 1 gather.go:177] gatherer "clusterconfig" function "crds" took 23.282573ms to process 2 records I0319 18:03:17.454249 1 prometheus_rules.go:88] Prometheus rules successfully created I0319 18:03:17.457542 1 tasks_processing.go:74] worker 11 stopped. I0319 18:03:17.460122 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-4lbnc with fingerprint=e81ba9ecdf1edf06d518d60e7ad727761018cacfefe4e85d80d2abe8a2e93f22 I0319 18:03:17.460581 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-s89qg with fingerprint=6d83b222b1146af06869863d85535b82b215e41ee6e8c285ad1f4961608d629b I0319 18:03:17.460680 1 recorder.go:75] Recording config/running_containers with fingerprint=6d29def28e545d76b7d750716b013fdfb8b84e7548708d0f3f72b5332c6fe232 I0319 18:03:17.460697 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 55.940687ms to process 3 records I0319 18:03:17.462437 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0319 18:03:17.464222 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:54204->172.30.0.10:53: read: connection refused E0319 18:03:17.464230 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2792f28552-5f52-4204-b7d7-fa8e0acf2bdc%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:54204->172.30.0.10:53: read: connection refused I0319 18:03:17.464239 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:54204->172.30.0.10:53: read: connection refused I0319 18:03:17.464243 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2792f28552-5f52-4204-b7d7-fa8e0acf2bdc%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:54204->172.30.0.10:53: read: connection refused I0319 18:03:17.468091 1 tasks_processing.go:74] worker 12 stopped. E0319 18:03:17.468111 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0319 18:03:17.468156 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0319 18:03:17.468164 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0319 18:03:17.468178 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=c8a9805405aae7d04b7f03c09c51097942070877009b6daebd5573ef8127446e I0319 18:03:17.468236 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0319 18:03:17.468247 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0319 18:03:17.468254 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=3b70b9787487b548204d806ebebf90b0c2d460145e34cb33c0d8d3a7a011d907 I0319 18:03:17.468261 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0319 18:03:17.468307 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0319 18:03:17.468318 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0319 18:03:17.468329 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 65.570366ms to process 7 records I0319 18:03:17.468946 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0319 18:03:17.469008 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 18:03:17.470012 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0319 18:03:17.472759 1 base_controller.go:82] Caches are synced for ConfigController I0319 18:03:17.472774 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0319 18:03:17.477409 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 18:03:17.512001 1 gather_logs.go:145] no pods in namespace were found I0319 18:03:17.512075 1 tasks_processing.go:74] worker 6 stopped. I0319 18:03:17.512115 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 96.385121ms to process 0 records I0319 18:03:17.512463 1 tasks_processing.go:74] worker 28 stopped. I0319 18:03:17.512815 1 recorder.go:75] Recording config/version with fingerprint=f55c29003433c6d0bd5ffacbbcfe65813f36a2adf637edadba5b3db515569110 I0319 18:03:17.512856 1 recorder.go:75] Recording config/id with fingerprint=7fd8f39dae63200289d00d1cfee99136f8f6d07283ed33fd38f9d48d93183b4d I0319 18:03:17.512886 1 gather.go:177] gatherer "clusterconfig" function "version" took 88.13549ms to process 2 records I0319 18:03:17.530699 1 tasks_processing.go:74] worker 15 stopped. I0319 18:03:17.530793 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0319 18:03:17.530830 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 124.812062ms to process 1 records I0319 18:03:17.535395 1 tasks_processing.go:74] worker 5 stopped. I0319 18:03:17.535470 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 122.413207ms to process 0 records I0319 18:03:17.537296 1 base_controller.go:82] Caches are synced for LoggingSyncer I0319 18:03:17.537314 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0319 18:03:17.543887 1 tasks_processing.go:74] worker 7 stopped. E0319 18:03:17.543933 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0319 18:03:17.543961 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p4uhd2dagqt1ot08javpdvdjktnsnmq-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p4uhd2dagqt1ot08javpdvdjktnsnmq-primary-cert-bundle-secret" not found I0319 18:03:17.544088 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=0f8380852e62f85157786d293ace06661768a14f74dc47d725c4425c8407d3cd I0319 18:03:17.544107 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 143.747182ms to process 1 records I0319 18:03:17.862624 1 gather_cluster_operator_pods_and_events.go:121] Found 16 pods with 18 containers I0319 18:03:17.862699 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1398101 bytes I0319 18:03:17.862745 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-grlh6 pod in namespace openshift-dns (previous: false). I0319 18:03:18.070949 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-grlh6 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-grlh6\" is waiting to start: ContainerCreating" I0319 18:03:18.070969 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-grlh6\" is waiting to start: ContainerCreating" I0319 18:03:18.070982 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-grlh6 pod in namespace openshift-dns (previous: false). I0319 18:03:18.274964 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-grlh6 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-grlh6\" is waiting to start: ContainerCreating" I0319 18:03:18.274985 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-grlh6\" is waiting to start: ContainerCreating" I0319 18:03:18.275002 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-pzfn4 pod in namespace openshift-dns (previous: false). W0319 18:03:18.441022 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 18:03:18.513000 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pzfn4 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-pzfn4\" is waiting to start: ContainerCreating" I0319 18:03:18.513083 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-pzfn4\" is waiting to start: ContainerCreating" I0319 18:03:18.513104 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-pzfn4 pod in namespace openshift-dns (previous: false). I0319 18:03:18.669121 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pzfn4 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-pzfn4\" is waiting to start: ContainerCreating" I0319 18:03:18.669151 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-pzfn4\" is waiting to start: ContainerCreating" I0319 18:03:18.669167 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-h6bzj pod in namespace openshift-dns (previous: false). I0319 18:03:18.692593 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 18:03:18.860092 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0319 18:03:18.870701 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:18.870719 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-m8rss pod in namespace openshift-dns (previous: false). I0319 18:03:19.109500 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:19.109522 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-tmtzf pod in namespace openshift-dns (previous: false). I0319 18:03:19.274627 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:19.274647 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-68cb65bb56-82lkv pod in namespace openshift-image-registry (previous: false). W0319 18:03:19.440080 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 18:03:19.462214 1 tasks_processing.go:74] worker 16 stopped. I0319 18:03:19.462261 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=42e08b9392c8b7b1a7e246d61cac64ae344aad6ff9d1d835339fe9696c019570 I0319 18:03:19.462303 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=499b37fa7ce906ed65ec6b07c49d05d6af203c48aa376bafbfb6977943620d34 I0319 18:03:19.462337 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0319 18:03:19.462372 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=65bfd6c3db0e660f2e7fce2697b87666b394157edf62b4a4a0ca22bd3c72c034 I0319 18:03:19.462411 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0319 18:03:19.462436 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=3265b01b20226764486ab63166c66e881022f9c902beb254867bba9a687c0f08 I0319 18:03:19.462473 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=e331ba0f80c0769bef314cc113746ff8432a10c0c3a2def4c62f892af27bf011 I0319 18:03:19.462500 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=98d6c42729861beecb9269deaf1d9c7750b75369359e5d26ec2b78c28a5eb575 I0319 18:03:19.462519 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=ee0721518acd2f821c5db901a39c1cc206ba51b359fbbf3b94c91c92fa2f6e53 I0319 18:03:19.462536 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=3477fbd390c1cf0aaf9197427c35d9f5f55c3021bf5bdbde7e1bcfc75dc72ceb I0319 18:03:19.462543 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0319 18:03:19.462558 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=d568ad884aaa92b8d9eda62b31d1ec9e8ab5538702506d2f5f9e5bde61ab0e7f I0319 18:03:19.462569 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0319 18:03:19.462584 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=9c09353352f032cf8f0fd36ff4d3cce41eda5fa2fe7bd8fef13d601d018992f6 I0319 18:03:19.462593 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0319 18:03:19.462607 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=839ed6affbe905d98966c6f512b1c2c30316c8e115eb67e52981dd7d588e423c I0319 18:03:19.462629 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0319 18:03:19.462657 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=c917af6ba2759fdf970c459a50ed3501572716a63915ffd98daa7db54ea70f23 I0319 18:03:19.462820 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=859d79f3548d5924c4129adf2e8d0a71a35e8936b40b1664a5380f9b3979852a I0319 18:03:19.462832 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0319 18:03:19.462839 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0319 18:03:19.462860 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0319 18:03:19.462885 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=2ffd7c8b821cadd5c5029a5fe5803c7b5b8f7677ac607a2ac317c5eb244f4ff8 I0319 18:03:19.462935 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=8135842f7a95d06bd8323dca0c483304502d94a41e68f83ecfc5dbde0e6a890b I0319 18:03:19.462947 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0319 18:03:19.462974 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=23466677f225092e34788cb449121f2fa46340bc3cd78433ecdfdd2642a9504d I0319 18:03:19.462989 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0319 18:03:19.463013 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=546c315148eaa766229fe91dfbc2d1b983bf483879d353986b9ffe2c0eff53bd I0319 18:03:19.463040 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=c343c97218803ccd46e3c7fb3552aac638d2833b595ba85b53f5cb8ad642f073 I0319 18:03:19.463068 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=122606b3181b55a56438529ffb9d35eb7212f710e40f89519e80ce72eb28ca29 I0319 18:03:19.463095 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=5dd7af0f69085c536e8e679ea031f1c6e83c7cc07aafbebc0c83289c7abfd410 I0319 18:03:19.463130 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=58e4a3144c08cddef75f8de06539dda10dac2864609084872f2a25b060c2019e I0319 18:03:19.463145 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0319 18:03:19.463184 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=f33d8e58d811e78a968f9193452276ee874a500cc882e17a92ca7405675582ae I0319 18:03:19.463214 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0319 18:03:19.463230 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0319 18:03:19.463242 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.062375787s to process 36 records I0319 18:03:19.467609 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-68cb65bb56-82lkv pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-68cb65bb56-82lkv\" is waiting to start: ContainerCreating" I0319 18:03:19.467628 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-68cb65bb56-82lkv\" is waiting to start: ContainerCreating" I0319 18:03:19.467641 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-68cb65bb56-n92qz pod in namespace openshift-image-registry (previous: false). I0319 18:03:19.672214 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-68cb65bb56-n92qz pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-68cb65bb56-n92qz\" is waiting to start: ContainerCreating" I0319 18:03:19.672235 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-68cb65bb56-n92qz\" is waiting to start: ContainerCreating" I0319 18:03:19.672248 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7fb9448555-fshwt pod in namespace openshift-image-registry (previous: false). I0319 18:03:19.862273 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:19.862296 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-2zsgz pod in namespace openshift-image-registry (previous: false). I0319 18:03:20.068847 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:20.068869 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-4fr8l pod in namespace openshift-image-registry (previous: false). I0319 18:03:20.270017 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:20.270034 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-wq8s2 pod in namespace openshift-image-registry (previous: false). W0319 18:03:20.438793 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 18:03:20.472912 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 18:03:20.472930 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-645cb7c6f8-4qxqd pod in namespace openshift-ingress (previous: false). I0319 18:03:20.668935 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-645cb7c6f8-4qxqd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-645cb7c6f8-4qxqd\" is waiting to start: ContainerCreating" I0319 18:03:20.668952 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-645cb7c6f8-4qxqd\" is waiting to start: ContainerCreating" I0319 18:03:20.668965 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-645cb7c6f8-wx9gb pod in namespace openshift-ingress (previous: false). I0319 18:03:20.867388 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-645cb7c6f8-wx9gb pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-645cb7c6f8-wx9gb\" is waiting to start: ContainerCreating" I0319 18:03:20.867408 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-645cb7c6f8-wx9gb\" is waiting to start: ContainerCreating" I0319 18:03:20.867429 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8899c9449-4ktqk pod in namespace openshift-ingress (previous: false). I0319 18:03:21.067514 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-8899c9449-4ktqk pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-8899c9449-4ktqk\" is waiting to start: ContainerCreating" I0319 18:03:21.067530 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-8899c9449-4ktqk\" is waiting to start: ContainerCreating" I0319 18:03:21.067542 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-jvq9l pod in namespace openshift-ingress-canary (previous: false). I0319 18:03:21.266945 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-rpr72 pod in namespace openshift-ingress-canary (previous: false). W0319 18:03:21.439268 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 18:03:21.469113 1 tasks_processing.go:74] worker 21 stopped. I0319 18:03:21.469200 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=347f60aa703fb759c973b8e1e1c4669dcebaca4d57f6b10966a65efaca324bf5 I0319 18:03:21.469245 1 recorder.go:75] Recording events/openshift-dns with fingerprint=651a6a597e3de9dd10c4bd3e44356ca13d6e16258e7ed02086a6714b187bd577 I0319 18:03:21.469328 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=16c362bb4f83b3bfc4e0a32b657e53d4545d2dc12b6818f90b8e6e6526dee1f9 I0319 18:03:21.469356 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=25cb1fb2ec2c282dbd66e56dc41b4987fab3a9fd28d61e17064aaa16b7b79b56 I0319 18:03:21.469401 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=8f2a1c9771894f6877eedce8b2e9b0f81e38db11a6cc172f0d28de957bce3635 I0319 18:03:21.469431 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=d4a452f0d009c7cab7223b8fdfdd4befcf2cc31bcd2af96d468bb07932180d7e I0319 18:03:21.469440 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-jvq9l/serve-healthcheck-canary_current.log with fingerprint=2cde3d6bb3fcd0d28f3f6b4c58ac2c4c4e688616b3ed9b17f57ecb0d888d65fa I0319 18:03:21.469446 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-rpr72/serve-healthcheck-canary_current.log with fingerprint=8675aab215713568c62632da742245074c83d55d3989b50a4bd92e8f3fba7a63 I0319 18:03:21.469455 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.034274454s to process 8 records W0319 18:03:22.439388 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 18:03:22.439412 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 18:03:22.439428 1 tasks_processing.go:74] worker 20 stopped. E0319 18:03:22.439438 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 18:03:22.439449 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0319 18:03:22.439462 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0319 18:03:22.439471 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.035963344s to process 1 records I0319 18:03:29.933993 1 tasks_processing.go:74] worker 1 stopped. I0319 18:03:29.934029 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0319 18:03:29.934042 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.529761968s to process 1 records I0319 18:03:30.611217 1 tasks_processing.go:74] worker 24 stopped. I0319 18:03:30.611463 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=7c1a3e3c431de69ce4ee6fb33aad2158a3d38446a91ec5899ca71ceaf725097e I0319 18:03:30.611479 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.20745871s to process 1 records E0319 18:03:30.611543 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.211s with: function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0319 18:03:30.612649 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0319 18:03:30.612662 1 periodic.go:216] Running workloads gatherer I0319 18:03:30.612677 1 tasks_processing.go:45] number of workers: 2 I0319 18:03:30.612682 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 18:03:30.612685 1 tasks_processing.go:71] worker 1 working on workload_info task. I0319 18:03:30.612697 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 18:03:30.612714 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0319 18:03:30.638133 1 tasks_processing.go:74] worker 0 stopped. I0319 18:03:30.638162 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.392527ms to process 0 records I0319 18:03:30.639614 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0319 18:03:30.654136 1 gather_workloads_info.go:387] No image sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575 (15ms) I0319 18:03:30.675253 1 gather_workloads_info.go:387] No image sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319 (21ms) I0319 18:03:30.692087 1 gather_workloads_info.go:387] No image sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6 (17ms) I0319 18:03:30.705165 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (13ms) I0319 18:03:30.727099 1 gather_workloads_info.go:387] No image sha256:0b2cc6aadb1abc1d35a55da0c9884eb9426d7ff7a97174daeb09bb1e1a23cfae (22ms) I0319 18:03:30.743284 1 gather_workloads_info.go:387] No image sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e (16ms) I0319 18:03:30.775156 1 gather_workloads_info.go:387] No image sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132 (32ms) I0319 18:03:30.786228 1 gather_workloads_info.go:387] No image sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b (11ms) I0319 18:03:30.799244 1 gather_workloads_info.go:387] No image sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd (13ms) I0319 18:03:30.816133 1 gather_workloads_info.go:387] No image sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c (17ms) I0319 18:03:30.832164 1 gather_workloads_info.go:387] No image sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6 (16ms) I0319 18:03:30.855233 1 gather_workloads_info.go:387] No image sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df (23ms) I0319 18:03:30.947771 1 gather_workloads_info.go:387] No image sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809 (93ms) I0319 18:03:31.047980 1 gather_workloads_info.go:387] No image sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635 (100ms) I0319 18:03:31.148335 1 gather_workloads_info.go:387] No image sha256:bd202c6a40d89b2766a77a53ab1d84feb452d9f5d01066e3b7318ed2a13f8d54 (100ms) I0319 18:03:31.248612 1 gather_workloads_info.go:387] No image sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574 (100ms) I0319 18:03:31.348827 1 gather_workloads_info.go:387] No image sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044 (100ms) I0319 18:03:31.448864 1 gather_workloads_info.go:387] No image sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211 (100ms) I0319 18:03:31.547896 1 gather_workloads_info.go:387] No image sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd (99ms) I0319 18:03:31.648256 1 gather_workloads_info.go:387] No image sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498 (100ms) I0319 18:03:31.748322 1 gather_workloads_info.go:387] No image sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648 (100ms) I0319 18:03:31.848547 1 gather_workloads_info.go:387] No image sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f (100ms) I0319 18:03:31.948629 1 gather_workloads_info.go:387] No image sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740 (100ms) I0319 18:03:32.048609 1 gather_workloads_info.go:387] No image sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a (100ms) I0319 18:03:32.152266 1 gather_workloads_info.go:387] No image sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958 (104ms) I0319 18:03:32.249642 1 gather_workloads_info.go:387] No image sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f (97ms) I0319 18:03:32.348726 1 gather_workloads_info.go:387] No image sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7 (99ms) I0319 18:03:32.448340 1 gather_workloads_info.go:387] No image sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d (100ms) I0319 18:03:32.548317 1 gather_workloads_info.go:387] No image sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae (100ms) I0319 18:03:32.648033 1 gather_workloads_info.go:387] No image sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a (100ms) I0319 18:03:32.748127 1 gather_workloads_info.go:387] No image sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329 (100ms) I0319 18:03:32.848258 1 gather_workloads_info.go:387] No image sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4 (100ms) I0319 18:03:32.951465 1 gather_workloads_info.go:387] No image sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc (103ms) I0319 18:03:33.048872 1 gather_workloads_info.go:387] No image sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb (97ms) I0319 18:03:33.148433 1 gather_workloads_info.go:387] No image sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004 (100ms) I0319 18:03:33.148465 1 tasks_processing.go:74] worker 1 stopped. E0319 18:03:33.148474 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0319 18:03:33.148739 1 recorder.go:75] Recording config/workload_info with fingerprint=31c5c37f3c83e71d936d91a8d4946e24793f7b71f674ad4dd24e3957c6c67f58 I0319 18:03:33.148757 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.535773319s to process 1 records E0319 18:03:33.148783 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.536s with: function \"workload_info\" failed with an error" I0319 18:03:33.149882 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0319 18:03:33.149895 1 periodic.go:216] Running conditional gatherer I0319 18:03:33.155004 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules I0319 18:03:33.161815 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.15:59429->172.30.0.10:53: read: connection refused E0319 18:03:33.162131 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 18:03:33.162190 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0319 18:03:33.167973 1 conditional_gatherer.go:392] cluster version is '4.21.5' E0319 18:03:33.167985 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.167989 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.167993 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.167996 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.167999 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.168001 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.168004 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.168008 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 18:03:33.168011 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0319 18:03:33.168024 1 tasks_processing.go:45] number of workers: 3 I0319 18:03:33.168033 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 18:03:33.168037 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0319 18:03:33.168047 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 18:03:33.168067 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 18:03:33.168082 1 tasks_processing.go:74] worker 1 stopped. I0319 18:03:33.168072 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0319 18:03:33.168081 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0319 18:03:33.168110 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0319 18:03:33.168121 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.062µs to process 1 records I0319 18:03:33.168150 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0319 18:03:33.168160 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 3.546µs to process 1 records I0319 18:03:33.168165 1 tasks_processing.go:74] worker 0 stopped. I0319 18:03:33.168358 1 tasks_processing.go:74] worker 2 stopped. I0319 18:03:33.168371 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 255.22µs to process 0 records I0319 18:03:33.168391 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.15:59429->172.30.0.10:53: read: connection refused I0319 18:03:33.168408 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0319 18:03:33.193740 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=56c6afd6c2237b399bc89408143ff7181ab3e5fdf172ec176532473985e232ac I0319 18:03:33.193869 1 diskrecorder.go:70] Writing 104 records to /var/lib/insights-operator/insights-2026-03-19-180333.tar.gz I0319 18:03:33.200257 1 diskrecorder.go:51] Wrote 104 records to disk in 6ms I0319 18:03:33.200291 1 periodic.go:285] Gathering cluster info every 2h0m0s I0319 18:03:33.200313 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0319 18:03:34.115873 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 18:03:34.315925 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 18:03:43.645671 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 18:04:36.284011 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="6a11bfa9e5ef734b0f5811486c7faa76872daac617970f690b81eab145d477e1") W0319 18:04:36.284040 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0319 18:04:36.284090 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="61c475b7739ff65a82524ff54f6c01f2ea477827cb314cb34336ccc0bee61f93") I0319 18:04:36.284092 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0319 18:04:36.284116 1 base_controller.go:181] Shutting down ConfigController ... I0319 18:04:36.284136 1 periodic.go:177] Shutting down I0319 18:04:36.284163 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0319 18:04:36.284159 1 base_controller.go:181] Shutting down LoggingSyncer ... I0319 18:04:36.284176 1 base_controller.go:113] All ConfigController workers have been terminated I0319 18:04:36.284177 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0319 18:04:36.284191 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="a304ab86c064f2d1047743043ae29a641e77c9f2955d52f967148d555edf5926") I0319 18:04:36.284191 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0319 18:04:36.284196 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0319 18:04:36.284205 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated"