W0703 21:59:54.295369 1 cmd.go:257] Using insecure, self-signed certificates I0703 21:59:54.602749 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 21:59:54.603093 1 observer_polling.go:159] Starting file observer I0703 21:59:55.413133 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0703 21:59:55.413427 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0703 21:59:55.413894 1 secure_serving.go:57] Forcing use of http/1.1 only W0703 21:59:55.413915 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0703 21:59:55.413919 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0703 21:59:55.413924 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0703 21:59:55.413927 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0703 21:59:55.413929 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0703 21:59:55.413931 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0703 21:59:55.414314 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0703 21:59:55.419704 1 secure_serving.go:213] Serving securely on [::]:8443 I0703 21:59:55.419722 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0703 21:59:55.419825 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0703 21:59:55.420110 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0703 21:59:55.419696 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"8bb6dad6-c8d3-4c06-9a4c-50db16b19a8a", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0703 21:59:55.419864 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-650648113/tls.crt::/tmp/serving-cert-650648113/tls.key" I0703 21:59:55.419862 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0703 21:59:55.420310 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0703 21:59:55.419887 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0703 21:59:55.419893 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0703 21:59:55.420477 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file W0703 21:59:55.484095 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0703 21:59:55.484133 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0703 21:59:55.484259 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0703 21:59:55.507487 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 21:59:55.507504 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0703 21:59:55.521012 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0703 21:59:55.521034 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0703 21:59:55.521059 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0703 21:59:55.524111 1 secretconfigobserver.go:119] support secret does not exist I0703 21:59:55.542464 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 21:59:55.550562 1 secretconfigobserver.go:119] support secret does not exist I0703 21:59:55.550897 1 recorder.go:161] Pruning old reports every 4h25m19s, max age is 288h0m0s I0703 21:59:55.558044 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0703 21:59:55.558057 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0703 21:59:55.558065 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0703 21:59:55.558072 1 insightsreport.go:296] Starting report retriever I0703 21:59:55.558080 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0703 21:59:55.558091 1 periodic.go:209] Running clusterconfig gatherer I0703 21:59:55.558128 1 tasks_processing.go:45] number of workers: 32 I0703 21:59:55.558154 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 21:59:55.558163 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 21:59:55.558166 1 tasks_processing.go:71] worker 1 working on dvo_metrics task. I0703 21:59:55.558169 1 tasks_processing.go:71] worker 0 working on validating_webhook_configurations task. I0703 21:59:55.558175 1 tasks_processing.go:69] worker 9 listening for tasks. I0703 21:59:55.558178 1 tasks_processing.go:69] worker 8 listening for tasks. I0703 21:59:55.558190 1 tasks_processing.go:69] worker 23 listening for tasks. I0703 21:59:55.558204 1 tasks_processing.go:69] worker 16 listening for tasks. I0703 21:59:55.558204 1 tasks_processing.go:69] worker 10 listening for tasks. I0703 21:59:55.558193 1 tasks_processing.go:69] worker 2 listening for tasks. I0703 21:59:55.558212 1 tasks_processing.go:69] worker 11 listening for tasks. I0703 21:59:55.558218 1 tasks_processing.go:69] worker 17 listening for tasks. I0703 21:59:55.558213 1 tasks_processing.go:69] worker 27 listening for tasks. I0703 21:59:55.558221 1 tasks_processing.go:69] worker 12 listening for tasks. I0703 21:59:55.558224 1 tasks_processing.go:69] worker 18 listening for tasks. I0703 21:59:55.558231 1 tasks_processing.go:69] worker 19 listening for tasks. I0703 21:59:55.558233 1 tasks_processing.go:69] worker 25 listening for tasks. I0703 21:59:55.558237 1 tasks_processing.go:69] worker 14 listening for tasks. I0703 21:59:55.558240 1 tasks_processing.go:69] worker 20 listening for tasks. I0703 21:59:55.558226 1 tasks_processing.go:69] worker 24 listening for tasks. I0703 21:59:55.558245 1 tasks_processing.go:69] worker 26 listening for tasks. I0703 21:59:55.558228 1 tasks_processing.go:69] worker 13 listening for tasks. I0703 21:59:55.558248 1 tasks_processing.go:69] worker 21 listening for tasks. I0703 21:59:55.558239 1 tasks_processing.go:69] worker 3 listening for tasks. I0703 21:59:55.558254 1 tasks_processing.go:69] worker 5 listening for tasks. I0703 21:59:55.558255 1 tasks_processing.go:69] worker 6 listening for tasks. I0703 21:59:55.558255 1 tasks_processing.go:69] worker 22 listening for tasks. I0703 21:59:55.558262 1 tasks_processing.go:69] worker 7 listening for tasks. I0703 21:59:55.558263 1 tasks_processing.go:69] worker 29 listening for tasks. I0703 21:59:55.558245 1 tasks_processing.go:69] worker 4 listening for tasks. I0703 21:59:55.558267 1 tasks_processing.go:69] worker 28 listening for tasks. I0703 21:59:55.558273 1 tasks_processing.go:69] worker 31 listening for tasks. I0703 21:59:55.558270 1 tasks_processing.go:69] worker 30 listening for tasks. I0703 21:59:55.558168 1 tasks_processing.go:69] worker 15 listening for tasks. I0703 21:59:55.558345 1 tasks_processing.go:71] worker 10 working on machine_healthchecks task. I0703 21:59:55.558350 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0703 21:59:55.558351 1 tasks_processing.go:71] worker 23 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0703 21:59:55.558355 1 tasks_processing.go:71] worker 9 working on storage_cluster task. I0703 21:59:55.558427 1 tasks_processing.go:71] worker 26 working on cost_management_metrics_configs task. I0703 21:59:55.558589 1 tasks_processing.go:71] worker 22 working on infrastructures task. I0703 21:59:55.558623 1 tasks_processing.go:71] worker 18 working on openshift_logging task. I0703 21:59:55.558665 1 tasks_processing.go:71] worker 21 working on ingress task. I0703 21:59:55.558712 1 tasks_processing.go:71] worker 6 working on node_logs task. I0703 21:59:55.558613 1 tasks_processing.go:71] worker 4 working on pod_network_connectivity_checks task. I0703 21:59:55.558826 1 tasks_processing.go:71] worker 28 working on openstack_version task. I0703 21:59:55.558623 1 tasks_processing.go:71] worker 29 working on overlapping_namespace_uids task. I0703 21:59:55.558870 1 tasks_processing.go:71] worker 27 working on storage_classes task. I0703 21:59:55.558618 1 tasks_processing.go:71] worker 7 working on operators task. I0703 21:59:55.559032 1 tasks_processing.go:71] worker 16 working on install_plans task. I0703 21:59:55.558640 1 tasks_processing.go:71] worker 14 working on certificate_signing_requests task. I0703 21:59:55.558648 1 tasks_processing.go:71] worker 25 working on machine_autoscalers task. I0703 21:59:55.558658 1 tasks_processing.go:71] worker 3 working on silenced_alerts task. W0703 21:59:55.559354 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 21:59:55.559387 1 tasks_processing.go:71] worker 3 working on cluster_apiserver task. I0703 21:59:55.558644 1 tasks_processing.go:71] worker 19 working on config_maps task. I0703 21:59:55.558662 1 tasks_processing.go:71] worker 13 working on clusterroles task. I0703 21:59:55.558674 1 tasks_processing.go:71] worker 24 working on openstack_dataplanedeployments task. I0703 21:59:55.558695 1 tasks_processing.go:71] worker 17 working on oauths task. I0703 21:59:55.558699 1 tasks_processing.go:71] worker 2 working on pdbs task. I0703 21:59:55.558703 1 tasks_processing.go:71] worker 11 working on schedulers task. I0703 21:59:55.558876 1 tasks_processing.go:71] worker 12 working on crds task. I0703 21:59:55.558887 1 tasks_processing.go:71] worker 30 working on container_images task. I0703 21:59:55.558892 1 tasks_processing.go:71] worker 31 working on machines task. I0703 21:59:55.558349 1 tasks_processing.go:71] worker 8 working on openshift_machine_api_events task. I0703 21:59:55.558685 1 tasks_processing.go:71] worker 5 working on sap_pods task. I0703 21:59:55.558669 1 tasks_processing.go:71] worker 20 working on jaegers task. I0703 21:59:55.559971 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 48.894µs to process 0 records I0703 21:59:55.563348 1 tasks_processing.go:71] worker 9 working on aggregated_monitoring_cr_names task. I0703 21:59:55.563357 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 4.976619ms to process 0 records I0703 21:59:55.563448 1 tasks_processing.go:71] worker 10 working on service_accounts task. E0703 21:59:55.563463 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0703 21:59:55.563479 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 5.086119ms to process 0 records I0703 21:59:55.566118 1 tasks_processing.go:71] worker 26 working on machine_configs task. I0703 21:59:55.566121 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 7.643139ms to process 0 records I0703 21:59:55.566277 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0703 21:59:55.566298 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0703 21:59:55.566307 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0703 21:59:55.566309 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0703 21:59:55.566325 1 controller.go:489] The operator is still being initialized I0703 21:59:55.566330 1 controller.go:512] The operator is healthy I0703 21:59:55.569846 1 tasks_processing.go:71] worker 18 working on machine_config_pools task. I0703 21:59:55.569885 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 11.172323ms to process 0 records I0703 21:59:55.577560 1 tasks_processing.go:71] worker 4 working on networks task. E0703 21:59:55.577585 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0703 21:59:55.577894 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 18.74222ms to process 0 records I0703 21:59:55.577930 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 18.68156ms to process 0 records I0703 21:59:55.577945 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 18.325635ms to process 0 records I0703 21:59:55.577951 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 18.057107ms to process 0 records I0703 21:59:55.577956 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 17.319655ms to process 0 records I0703 21:59:55.577968 1 tasks_processing.go:71] worker 25 working on authentication task. I0703 21:59:55.578122 1 tasks_processing.go:71] worker 0 working on image task. I0703 21:59:55.578153 1 tasks_processing.go:71] worker 28 working on sap_config task. I0703 21:59:55.578137 1 tasks_processing.go:71] worker 5 working on nodenetworkstates task. I0703 21:59:55.578143 1 tasks_processing.go:71] worker 24 working on image_pruners task. I0703 21:59:55.578243 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=940124449e36300756b5bc5540381529efe9b37cb3f8cbe9bb2da8b8e4d7e4f2 I0703 21:59:55.578584 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=a56c7bc425056551c5f957514452939d52df4a975c2dfe6b54c2faa8d7f90203 I0703 21:59:55.578621 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=50c548e415a9d5f269fddc0c9c4cf030d8197e0e1a906b8294d99667c90f4bd0 I0703 21:59:55.578660 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=6c108dbccf5c1ab9e644c0d89b4b373af5fa2e340a976d3c4fa827a82d544fd1 I0703 21:59:55.578725 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=c94ae0672a3489c347ba970fc100707314d3556eba7bba78f18ddd4d9ec2aba9 I0703 21:59:55.578761 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=7765a5534d1aa543c097602f899e1ddc00aba4356609fc36e2f565a5b16b7f65 I0703 21:59:55.578823 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=524aa1e2ba9a7a27f4dc0c650c745bad7d4823cc9516539b22b654e4423a0aa1 I0703 21:59:55.578875 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=0e7ddeb016489ec49095d8c5e89689d38fa672f408b66a87d01559f82b305a60 I0703 21:59:55.578913 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=3d297361985f26c8f9afd76235d68030880c3d00df40a03d39e778bfb64f3913 I0703 21:59:55.578948 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=f0ecd04ec9d7b2374ccdbd8d64dd44ec71bc60f4010c528ea643457297f3b4cb I0703 21:59:55.578982 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=35e1eabc10da54d2837234bc0f2274e6e0d57efa0dc29ac175b2b09ecab346c5 I0703 21:59:55.578994 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 19.676892ms to process 11 records I0703 21:59:55.579097 1 tasks_processing.go:71] worker 21 working on qemu_kubevirt_launcher_logs task. I0703 21:59:55.579205 1 recorder.go:75] Recording config/ingress with fingerprint=630ae261e9be7ee49ec2f7da097e018cd0ed374992a9c35dc8eb29aeeca76616 I0703 21:59:55.579220 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 19.246023ms to process 1 records I0703 21:59:55.579309 1 tasks_processing.go:71] worker 22 working on image_registries task. I0703 21:59:55.579896 1 recorder.go:75] Recording config/infrastructure with fingerprint=a06e25271c527c82a111bb40a77c151605ce4e4ab4c716e4dfe38353abe71810 I0703 21:59:55.579917 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 19.381096ms to process 1 records I0703 21:59:55.580014 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=09381072e660d0d983ef95637f6a3b50e56c9a66c22ef55e5553a8b61272acef I0703 21:59:55.580018 1 tasks_processing.go:71] worker 27 working on nodenetworkconfigurationpolicies task. I0703 21:59:55.580044 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=5e9f11f3fa73b3ed4a566f0eeb731cb15dca9e6c4d21879bcce7174aa6a1faac I0703 21:59:55.580057 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 19.20339ms to process 2 records E0703 21:59:55.580066 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0703 21:59:55.580076 1 gather.go:177] gatherer "clusterconfig" function "machines" took 17.888347ms to process 0 records I0703 21:59:55.580083 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 17.8616ms to process 0 records I0703 21:59:55.580160 1 tasks_processing.go:71] worker 31 working on monitoring_persistent_volumes task. I0703 21:59:55.580275 1 tasks_processing.go:71] worker 20 working on openstack_dataplanenodesets task. W0703 21:59:55.583382 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0703 21:59:55.584428 1 base_controller.go:82] Caches are synced for ConfigController I0703 21:59:55.584438 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0703 21:59:55.596996 1 tasks_processing.go:71] worker 8 working on sap_datahubs task. I0703 21:59:55.597008 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 36.590523ms to process 0 records I0703 21:59:55.603296 1 tasks_processing.go:71] worker 3 working on nodes task. I0703 21:59:55.603480 1 recorder.go:75] Recording config/apiserver with fingerprint=9f567876e2421dfec3f487ae9b7296d7ccc26f960ed3b524aa1f696e426120ed I0703 21:59:55.603503 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 43.887906ms to process 1 records I0703 21:59:55.603691 1 tasks_processing.go:71] worker 17 working on feature_gates task. I0703 21:59:55.603769 1 recorder.go:75] Recording config/oauth with fingerprint=05f49874bbd3cdb0b1bf4533b147d212436fa97c7ded7d02a85dd62b771c2305 I0703 21:59:55.603839 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 43.486688ms to process 1 records I0703 21:59:55.603854 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 23.438803ms to process 0 records I0703 21:59:55.603860 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 45.094679ms to process 0 records I0703 21:59:55.603868 1 tasks_processing.go:71] worker 31 working on machine_sets task. I0703 21:59:55.603936 1 tasks_processing.go:71] worker 6 working on mutating_webhook_configurations task. I0703 21:59:55.604117 1 tasks_processing.go:71] worker 20 working on container_runtime_configs task. I0703 21:59:55.604240 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 23.82861ms to process 0 records I0703 21:59:55.604819 1 tasks_processing.go:71] worker 2 working on proxies task. I0703 21:59:55.604934 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=d7a269fd9c4c58a77839c643985982bbcc1ce831ac9c6b830fca828e5bdd1047 I0703 21:59:55.604966 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=ca19f93442d5c87ab0324b9a5c6a81402fba609a6b873ad7677514e06ca8ecf8 I0703 21:59:55.604990 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e41d845df8dcde761f3b63f9dbefd783204740cfb30c85a824f1794410bc9487 I0703 21:59:55.605000 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 44.781297ms to process 3 records I0703 21:59:55.605063 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=932213be07af601c0d7f2f7ac41fe20d611901e8b76290c6f2fb463e358ccc0c I0703 21:59:55.605111 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 44.924779ms to process 1 records I0703 21:59:55.605142 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 26.772371ms to process 0 records I0703 21:59:55.605173 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 26.852498ms to process 0 records I0703 21:59:55.605198 1 tasks_processing.go:71] worker 11 working on ceph_cluster task. I0703 21:59:55.605332 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 25.296894ms to process 0 records I0703 21:59:55.605503 1 tasks_processing.go:71] worker 4 working on tsdb_status task. I0703 21:59:55.605560 1 recorder.go:75] Recording config/network with fingerprint=63bacdb4416eaa58ff4a126fbb8f0105fea4c014def799c1eb2fe2a508f8df51 I0703 21:59:55.605858 1 gather.go:177] gatherer "clusterconfig" function "networks" took 27.701402ms to process 1 records I0703 21:59:55.606050 1 recorder.go:75] Recording config/image with fingerprint=c759d5e04808772141a583f970ee0cc96226ab7f3a25a0bbfedb48a780fcef55 I0703 21:59:55.606102 1 gather.go:177] gatherer "clusterconfig" function "image" took 27.304052ms to process 1 records I0703 21:59:55.606131 1 tasks_processing.go:71] worker 0 working on olm_operators task. W0703 21:59:55.605573 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 21:59:55.605520 1 tasks_processing.go:71] worker 5 working on support_secret task. I0703 21:59:55.605525 1 tasks_processing.go:71] worker 28 working on active_alerts task. W0703 21:59:55.606524 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 21:59:55.605535 1 tasks_processing.go:71] worker 27 working on metrics task. W0703 21:59:55.606613 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 21:59:55.605507 1 gather_logs.go:145] no pods in namespace were found I0703 21:59:55.606810 1 recorder.go:75] Recording config/authentication with fingerprint=21ad139e5a0c943475032679e94bbba012534d792612a60c9eeebb9d43125b55 I0703 21:59:55.606828 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 27.613394ms to process 1 records I0703 21:59:55.606838 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 9.165964ms to process 0 records I0703 21:59:55.606920 1 tasks_processing.go:71] worker 8 working on lokistack task. I0703 21:59:55.607378 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=3ade7cd8377e954db837237f25c56c4088495719e0d2cd6dbcb9dd825ce8b513 I0703 21:59:55.607438 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 27.865087ms to process 1 records I0703 21:59:55.607468 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 774.281µs to process 0 records I0703 21:59:55.607479 1 tasks_processing.go:71] worker 25 working on version task. I0703 21:59:55.607489 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 82.301µs to process 0 records I0703 21:59:55.607497 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 81.348µs to process 0 records I0703 21:59:55.607504 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 27.518293ms to process 0 records I0703 21:59:55.607514 1 tasks_processing.go:71] worker 21 working on ingress_certificates task. I0703 21:59:55.607399 1 tasks_processing.go:71] worker 24 working on openstack_controlplanes task. I0703 21:59:55.608053 1 tasks_processing.go:74] worker 4 stopped. I0703 21:59:55.608053 1 tasks_processing.go:74] worker 28 stopped. I0703 21:59:55.608103 1 tasks_processing.go:74] worker 27 stopped. I0703 21:59:55.609920 1 tasks_processing.go:74] worker 11 stopped. I0703 21:59:55.610115 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 4.66066ms to process 0 records I0703 21:59:55.610157 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 6.104617ms to process 0 records I0703 21:59:55.610202 1 tasks_processing.go:74] worker 31 stopped. I0703 21:59:55.610432 1 tasks_processing.go:74] worker 22 stopped. I0703 21:59:55.610666 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=15efb48b13370653dc2835929223f9affded2698eb6a0321864169cae03e45ef I0703 21:59:55.610705 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 30.695407ms to process 1 records I0703 21:59:55.610741 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0703 21:59:55.610800 1 tasks_processing.go:74] worker 29 stopped. I0703 21:59:55.610805 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 51.599112ms to process 1 records I0703 21:59:55.611399 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 6.63098ms to process 0 records I0703 21:59:55.611491 1 tasks_processing.go:74] worker 20 stopped. I0703 21:59:55.611559 1 tasks_processing.go:74] worker 12 stopped. I0703 21:59:55.612139 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=6c13b1eb1445505e5ce8e3ad7a62a430ae5747270bb4b8dcca3a3e69f251f32f I0703 21:59:55.612455 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=cc949da441c09ae61d0f7ca34ca7d63ef978ca504ab18aebeafcc67c4bc676be I0703 21:59:55.612477 1 gather.go:177] gatherer "clusterconfig" function "crds" took 50.679074ms to process 2 records I0703 21:59:55.612488 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.546584ms to process 0 records I0703 21:59:55.612498 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 53.209536ms to process 0 records I0703 21:59:55.612506 1 tasks_processing.go:74] worker 14 stopped. I0703 21:59:55.612511 1 tasks_processing.go:74] worker 8 stopped. I0703 21:59:55.613658 1 tasks_processing.go:74] worker 5 stopped. E0703 21:59:55.613690 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0703 21:59:55.613702 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 7.304957ms to process 0 records I0703 21:59:55.616009 1 tasks_processing.go:74] worker 24 stopped. I0703 21:59:55.616026 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 7.934968ms to process 0 records I0703 21:59:55.616096 1 tasks_processing.go:74] worker 13 stopped. I0703 21:59:55.616331 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=b80fbc37e9e057f32596cfe4ee093c21b14ab561fb28f32ebb60a35588eb0e6b I0703 21:59:55.616420 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=35aaabcc80a69d772f160ce95e8235271690a0cb79fb3c3672070916b922332a I0703 21:59:55.616428 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 56.611328ms to process 2 records I0703 21:59:55.616491 1 tasks_processing.go:74] worker 17 stopped. I0703 21:59:55.616603 1 recorder.go:75] Recording config/featuregate with fingerprint=b3215c9f7c9702f2ad6c2b0793ef120dba1be35d762042f133da0a86d3571c70 I0703 21:59:55.616617 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 12.778305ms to process 1 records I0703 21:59:55.619188 1 tasks_processing.go:74] worker 2 stopped. I0703 21:59:55.619262 1 recorder.go:75] Recording config/proxy with fingerprint=a587bf71ba3b4befc182443da41f86a5e800758940c3473d3bea73bdf730da12 I0703 21:59:55.619276 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 14.349039ms to process 1 records I0703 21:59:55.619403 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0703 21:59:55.619471 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0703 21:59:55.619525 1 operator.go:288] started I0703 21:59:55.619553 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0703 21:59:55.624387 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 21:59:55.624405 1 tasks_processing.go:74] worker 9 stopped. I0703 21:59:55.624419 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 61.034676ms to process 0 records I0703 21:59:55.625092 1 tasks_processing.go:74] worker 6 stopped. I0703 21:59:55.625355 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=3542d1f39932019aad0a6e0d7d258d92ea873b921a48d70d654cad6a15fb304a I0703 21:59:55.625406 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=24b386f41c18d20e54c3963a9ac3dfdaa5ea540e1887c4088c54a2acf414f282 I0703 21:59:55.625447 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=845b25e2907383ac3508fab43c6cab5db3471c010b2c6c771da09d8460860438 I0703 21:59:55.625497 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 21.107707ms to process 3 records I0703 21:59:55.627698 1 tasks_processing.go:74] worker 30 stopped. I0703 21:59:55.630931 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-lkd9t with fingerprint=b204a6619bd3ac39e8e2d914d965861cb46b1c0cf330e40b1dadf63e33e27d83 I0703 21:59:55.631181 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-5lmtd with fingerprint=61bd3063563bdfe5153596a134a34e23c40612a111144d633c0d7e965a391d18 I0703 21:59:55.631397 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-gcp2n with fingerprint=79f7180f8cfb5b78ed119b48005a749868ca008694e29be9b271c154f8f4819c I0703 21:59:55.631447 1 recorder.go:75] Recording config/running_containers with fingerprint=44128d97c8887d3897b26ff4e6db39670575783ab1c0a89ea270efaf0a081d64 I0703 21:59:55.631456 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 67.371686ms to process 4 records I0703 21:59:55.638507 1 tasks_processing.go:74] worker 23 stopped. I0703 21:59:55.638547 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 80.137189ms to process 0 records I0703 21:59:55.639113 1 tasks_processing.go:74] worker 0 stopped. I0703 21:59:55.639147 1 prometheus_rules.go:88] Prometheus rules successfully created I0703 21:59:55.639214 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0703 21:59:55.639223 1 recorder.go:75] Recording config/olm_operators with fingerprint=0345b999b00538e8973039585483186087f9c452a0cadfd6e7711796b5ec5ebc I0703 21:59:55.639228 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0703 21:59:55.639234 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0703 21:59:55.639237 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0703 21:59:55.639241 1 controller.go:212] Source scaController *sca.Controller is not ready I0703 21:59:55.639240 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 32.92646ms to process 1 records I0703 21:59:55.639256 1 controller.go:489] The operator is still being initialized I0703 21:59:55.639261 1 controller.go:512] The operator is healthy E0703 21:59:55.647595 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27c7204458-122f-48a3-accc-aad18925e513%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:60672->172.30.0.10:53: read: connection refused I0703 21:59:55.647612 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27c7204458-122f-48a3-accc-aad18925e513%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:60672->172.30.0.10:53: read: connection refused I0703 21:59:55.649748 1 tasks_processing.go:74] worker 19 stopped. E0703 21:59:55.649765 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0703 21:59:55.649773 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0703 21:59:55.649796 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0703 21:59:55.649805 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0703 21:59:55.649834 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0703 21:59:55.649841 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0703 21:59:55.649845 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0703 21:59:55.649849 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0703 21:59:55.649885 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0703 21:59:55.649892 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0703 21:59:55.649897 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 90.319415ms to process 7 records I0703 21:59:55.656896 1 tasks_processing.go:74] worker 21 stopped. E0703 21:59:55.656911 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0703 21:59:55.656918 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ratrkdkluad89chjacv8i03daia4uc2-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ratrkdkluad89chjacv8i03daia4uc2-primary-cert-bundle-secret" not found I0703 21:59:55.656973 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=a3f7660a3e49afd952aa62fb7fd24b4ca562e96382877a8787ca22e507763744 I0703 21:59:55.656989 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 49.36949ms to process 1 records I0703 21:59:55.676615 1 tasks_processing.go:74] worker 18 stopped. I0703 21:59:55.676638 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 106.750443ms to process 0 records I0703 21:59:55.678430 1 tasks_processing.go:74] worker 3 stopped. I0703 21:59:55.678723 1 recorder.go:75] Recording config/node/ip-10-0-1-217.ec2.internal with fingerprint=8861d885b0cac27a1ef82533a33d349b4daee43e2a14bc72aeca44ca82d8942a I0703 21:59:55.678800 1 recorder.go:75] Recording config/node/ip-10-0-2-97.ec2.internal with fingerprint=c60d1b85bddd88c848dd85be702d1ae117fcd9ecb654678e9df1caf926c57b67 I0703 21:59:55.678814 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 75.074671ms to process 2 records I0703 21:59:55.679091 1 tasks_processing.go:74] worker 25 stopped. I0703 21:59:55.679361 1 recorder.go:75] Recording config/version with fingerprint=c12ee3b8ac0e65f5e05c5d2ef1eb69978291fda3ee05526744b5af93d256cc16 I0703 21:59:55.679373 1 recorder.go:75] Recording config/id with fingerprint=c555e7fb0179cd33c58984dea5b4ce9aba52d53d3559e1aecca22bc7e52ce95e I0703 21:59:55.679378 1 gather.go:177] gatherer "clusterconfig" function "version" took 71.595667ms to process 2 records I0703 21:59:55.681653 1 tasks_processing.go:74] worker 26 stopped. I0703 21:59:55.681677 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0703 21:59:55.681684 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 115.516241ms to process 1 records I0703 21:59:55.692288 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0703 21:59:55.695761 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:44729->172.30.0.10:53: read: connection refused I0703 21:59:55.695773 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:44729->172.30.0.10:53: read: connection refused I0703 21:59:55.720233 1 base_controller.go:82] Caches are synced for LoggingSyncer I0703 21:59:55.720253 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... W0703 21:59:56.581751 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0703 21:59:56.788011 1 gather_cluster_operator_pods_and_events.go:121] Found 16 pods with 19 containers I0703 21:59:56.788024 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1324517 bytes I0703 21:59:56.788061 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-dclsb pod in namespace openshift-dns (previous: false). I0703 21:59:57.010578 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-dclsb pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-dclsb\" is waiting to start: ContainerCreating" I0703 21:59:57.010595 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-dclsb\" is waiting to start: ContainerCreating" I0703 21:59:57.010604 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-dclsb pod in namespace openshift-dns (previous: false). I0703 21:59:57.046707 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0703 21:59:57.191387 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-dclsb pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-dclsb\" is waiting to start: ContainerCreating" I0703 21:59:57.191403 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-dclsb\" is waiting to start: ContainerCreating" I0703 21:59:57.191431 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-gpb72 pod in namespace openshift-dns (previous: false). I0703 21:59:57.409135 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gpb72 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-gpb72\" is waiting to start: ContainerCreating" I0703 21:59:57.409154 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-gpb72\" is waiting to start: ContainerCreating" I0703 21:59:57.409166 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-gpb72 pod in namespace openshift-dns (previous: false). I0703 21:59:57.448954 1 tasks_processing.go:74] worker 7 stopped. I0703 21:59:57.449002 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=100e72190604b760b24ab39b1f0148bf03b38169bfe3491035f78bcf95dc0c10 I0703 21:59:57.449041 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=dd0d55b05bb7eb7b95418a9e41aac85591a3efb50da3eb82f3d98351a95d2bc0 I0703 21:59:57.449070 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0703 21:59:57.449112 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=21dd06140c1ddd2e9488c8a8037689932b698e359a1f4bc8e0943b18281ec748 I0703 21:59:57.449129 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0703 21:59:57.449157 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=2e6006cddba1d09858c32741d69ab8a4fac6ce2d378bfcbd8b868ad0b99b445f I0703 21:59:57.449189 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=8755d2c18617a8dd4f37b2362b42cf204d98eb9cc2d4ba82596118f7d9e54bd5 I0703 21:59:57.449212 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=90fb1006b3454c585db02c2502732177ad2bc77dfaade776c91f107964a5c64e I0703 21:59:57.449226 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=452be6565e5acea35680b6f42509de05363e08fe67105492cd0a4e80a245f87c I0703 21:59:57.449245 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=e49549b2c7274d7d537126f200652a7d52affbed84d07f3ffee42206b85a4345 I0703 21:59:57.449254 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0703 21:59:57.449270 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=64fc286dcfd3fa7f7eb4ac1a3a1f71f3a2b689d8a87ef3eb54799a30fe03eab2 I0703 21:59:57.449278 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0703 21:59:57.449294 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=625c65e62162139a2923b33750e37071fe97e8aa243180ae25e4334cea123ab8 I0703 21:59:57.449303 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0703 21:59:57.449324 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c8618c0929fef6368ac9083919a2ae70990ce83f225a89806897ad5134bf8c64 I0703 21:59:57.449333 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0703 21:59:57.449348 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=2eb5068d7f81409c0f9147739419c5bf395335f00bfa8e94aef91a16b6bce9b0 I0703 21:59:57.449469 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=5550f0f0cdfe886849d3e257e76df24bd62466509324015eeb7eb5b4a0ca915c I0703 21:59:57.449479 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0703 21:59:57.449486 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0703 21:59:57.449511 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0703 21:59:57.449532 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=f26b26d5fbf4e134482105a6e2cda6bbdfb65a8ac90f2acd2dd3ae7fd8472760 I0703 21:59:57.449555 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=ede1723149bb6219aeff51c14fa89b8a94dd21dd0c661fcf03c2c9f585976878 I0703 21:59:57.449566 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0703 21:59:57.449581 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=018fef1ebff04f2c3771a1fe6ae13d85e75c3aa8bf6e10752958961cb2d7f3a0 I0703 21:59:57.449588 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0703 21:59:57.449600 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=bb6f12adb340ea159494ec3703da78e276c52082c5f0ee8e7b2e6550ccdfc825 I0703 21:59:57.449614 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=f651bfa5d071d1971fba44157c36e7ace8822c0da45a8ea8a23058d0b5dbb3b4 I0703 21:59:57.449628 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=d6dda5fe108230b1853c956b681f145c6cbf7ca71da3b4d3bd84b75bbdeab630 I0703 21:59:57.449643 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=49d37ad0cf081dec7f29442a489f59c8bca3bc288ba82d77b68eb3635e19f997 I0703 21:59:57.449654 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=f1de0b4b66c61bd307ae12283dce8a0c157cae5f5edffff14826118bd302e8ed I0703 21:59:57.449684 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b87c0c8c73ec9f60bbefe5ddfaf716e12ffaf6f158a6b12499f6363147ad31da I0703 21:59:57.449700 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0703 21:59:57.449709 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0703 21:59:57.449716 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.889928696s to process 35 records W0703 21:59:57.581960 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0703 21:59:57.589986 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gpb72 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-gpb72\" is waiting to start: ContainerCreating" I0703 21:59:57.589999 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-gpb72\" is waiting to start: ContainerCreating" I0703 21:59:57.590009 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-5txwx pod in namespace openshift-dns (previous: false). I0703 21:59:57.790926 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0703 21:59:57.790940 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-5vllm pod in namespace openshift-dns (previous: false). I0703 21:59:57.991871 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0703 21:59:57.991962 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5797569868-6n4hl pod in namespace openshift-image-registry (previous: false). I0703 21:59:58.186103 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0703 21:59:58.186150 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5797569868-vn648 pod in namespace openshift-image-registry (previous: false). I0703 21:59:58.394125 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-5797569868-vn648 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-5797569868-vn648\" is waiting to start: ContainerCreating" I0703 21:59:58.394140 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-5797569868-vn648\" is waiting to start: ContainerCreating" I0703 21:59:58.394183 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-684c78b5c5-pjxcr pod in namespace openshift-image-registry (previous: false). W0703 21:59:58.581933 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0703 21:59:58.591858 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-684c78b5c5-pjxcr pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-684c78b5c5-pjxcr\" is waiting to start: ContainerCreating" I0703 21:59:58.591872 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-684c78b5c5-pjxcr\" is waiting to start: ContainerCreating" I0703 21:59:58.591882 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-gk6r5 pod in namespace openshift-image-registry (previous: false). I0703 21:59:58.791679 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0703 21:59:58.791695 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-rzxvv pod in namespace openshift-image-registry (previous: false). I0703 21:59:58.993073 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0703 21:59:58.993088 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-78bfc5dfb9-lqdds pod in namespace openshift-ingress (previous: false). I0703 21:59:59.191597 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-78bfc5dfb9-lqdds pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-78bfc5dfb9-lqdds\" is waiting to start: ContainerCreating" I0703 21:59:59.191613 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-78bfc5dfb9-lqdds\" is waiting to start: ContainerCreating" I0703 21:59:59.191622 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-78bfc5dfb9-m2b2l pod in namespace openshift-ingress (previous: false). I0703 21:59:59.391815 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-78bfc5dfb9-m2b2l pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-78bfc5dfb9-m2b2l\" is waiting to start: ContainerCreating" I0703 21:59:59.391829 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-78bfc5dfb9-m2b2l\" is waiting to start: ContainerCreating" I0703 21:59:59.391838 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8678cb65f9-58rcz pod in namespace openshift-ingress (previous: false). W0703 21:59:59.582177 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0703 21:59:59.590447 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-8678cb65f9-58rcz pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-8678cb65f9-58rcz\" is waiting to start: ContainerCreating" I0703 21:59:59.590461 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-8678cb65f9-58rcz\" is waiting to start: ContainerCreating" I0703 21:59:59.590489 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-5r5bz pod in namespace openshift-ingress-canary (previous: false). I0703 21:59:59.791487 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-5r5bz pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5r5bz\" is waiting to start: ContainerCreating" I0703 21:59:59.791504 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5r5bz\" is waiting to start: ContainerCreating" I0703 21:59:59.791532 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-ldvxv pod in namespace openshift-ingress-canary (previous: false). I0703 21:59:59.991372 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-ldvxv pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-ldvxv\" is waiting to start: ContainerCreating" I0703 21:59:59.991389 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-ldvxv\" is waiting to start: ContainerCreating" I0703 21:59:59.991402 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-c5xjm pod in namespace openshift-kube-storage-version-migrator (previous: false). I0703 22:00:00.193293 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-c5xjm pod in namespace openshift-kube-storage-version-migrator (previous: false). I0703 22:00:00.391212 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-5c6fz pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). W0703 22:00:00.578405 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded E0703 22:00:00.578445 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0703 22:00:00.578463 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0703 22:00:00.578462 1 tasks_processing.go:74] worker 1 stopped. W0703 22:00:00.578485 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0703 22:00:00.578502 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.020250552s to process 1 records I0703 22:00:00.593150 1 tasks_processing.go:74] worker 15 stopped. I0703 22:00:00.593253 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=e1c30b05c6d45ab56f9d52847faa6720aa1a87b061a073eac60111acc6cf4661 I0703 22:00:00.593294 1 recorder.go:75] Recording events/openshift-dns with fingerprint=e7f36292b3f78afcbde7ccda8c1d992e4304900ba23e0c17fa58f717192ba915 I0703 22:00:00.593381 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=81585358e1f7a6b035eb6bf2f38aa0e488b3c0b727abd2460d801203c575b031 I0703 22:00:00.593415 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=da7a9e378a48fe7b0c3fa6f61be1e9b8f02c2233ca35c6dfdcf20d954d65d84c I0703 22:00:00.593461 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=fd6909bbc9d930b3f36bd08b3e09778f5f1e2a99f0e5f96036c1ea809db323f6 I0703 22:00:00.593474 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=238df4ac9843b84b6ebe413a85c71fc093e5c07d403dd5eaf33067292c159016 I0703 22:00:00.593485 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=381423c7c0af32cc03cccf57af1dc9f7714d12397435d7d3094fe831a17ec7e7 I0703 22:00:00.593556 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=47abaf4d88da455649aace0afebd022653b4b4eeeb05b32aba0e103e0142513b I0703 22:00:00.593693 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-dclsb with fingerprint=1488f2ef4c849d73117b8ba206f32e7f9567d928671813a6b0bfcdb9d8b21a6c I0703 22:00:00.593768 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-gpb72 with fingerprint=ef6c89f42922c33447f0726e44206dc51abd210844b2a17d50cbad3aa8389c69 I0703 22:00:00.593875 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5797569868-6n4hl with fingerprint=bfd42715221ba0e44bbcc0269e976407c5eae134279c3e6516c5a8a0b7aba860 I0703 22:00:00.593969 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5797569868-vn648 with fingerprint=6afc8d64e187a354d5ef4b3c52a88fe065d1bd2b2540fb26ef89302a49e160d5 I0703 22:00:00.594082 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-684c78b5c5-pjxcr with fingerprint=5f1f31618565d03092573bdc42e3a7751153c58c0b2ce3c7acab94a910b39a54 I0703 22:00:00.594140 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-5r5bz with fingerprint=a2e70637b3548a4e97c7d1c1649c62f39704899460235a2d012e901720037c72 I0703 22:00:00.594195 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-ldvxv with fingerprint=c8ca7cb62325ce69f668522c4b66b8ac311663ae294ab9a26935e6c9ca853533 I0703 22:00:00.594211 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-c5xjm/migrator_current.log with fingerprint=8efdabd2f48a87ba7bdf4cfe67e5328e444b072be9bbe791f6039c2a7d17b549 I0703 22:00:00.594215 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-c5xjm/graceful-termination_current.log with fingerprint=4aac520ed071a0723b41f5ef439526392330b1175e3d51c317cc076bb13b4604 I0703 22:00:00.594286 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-5c6fz/kube-storage-version-migrator-operator_current.log with fingerprint=c27399e993a16d6d446a9b34dde4dbdda9e93d3dba556d00404cd763b5da5009 I0703 22:00:00.594298 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 5.034785343s to process 18 records I0703 22:00:08.416253 1 tasks_processing.go:74] worker 16 stopped. I0703 22:00:08.416304 1 recorder.go:75] Recording config/installplans with fingerprint=e3c18441424aaa81f54ea9723d194eacb5824eb47178047bd893de11825bc8a2 I0703 22:00:08.416319 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.857158115s to process 1 records I0703 22:00:08.605224 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:00:09.168501 1 tasks_processing.go:74] worker 10 stopped. I0703 22:00:09.168808 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=c5e0ad81555672a1ae6992800f0e8a9ad0a7406d6b5e671cb8278224bb66f9c4 I0703 22:00:09.168831 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605030635s to process 1 records E0703 22:00:09.168900 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.61s with: function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0703 22:00:09.170008 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0703 22:00:09.170024 1 periodic.go:209] Running workloads gatherer I0703 22:00:09.170038 1 tasks_processing.go:45] number of workers: 2 I0703 22:00:09.170044 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 22:00:09.170048 1 tasks_processing.go:71] worker 1 working on workload_info task. I0703 22:00:09.170058 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 22:00:09.170070 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0703 22:00:09.191991 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 20s for image data I0703 22:00:09.195177 1 tasks_processing.go:74] worker 0 stopped. I0703 22:00:09.195192 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.093852ms to process 0 records I0703 22:00:09.200352 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (9ms) I0703 22:00:09.209079 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (9ms) I0703 22:00:09.217084 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (8ms) I0703 22:00:09.229645 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (13ms) I0703 22:00:09.238006 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (8ms) I0703 22:00:09.245918 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0703 22:00:09.254381 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (8ms) I0703 22:00:09.265275 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (11ms) I0703 22:00:09.273470 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (8ms) I0703 22:00:09.281759 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (8ms) I0703 22:00:09.301151 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (19ms) I0703 22:00:09.399961 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (99ms) I0703 22:00:09.500225 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0703 22:00:09.599927 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0703 22:00:09.700641 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (101ms) I0703 22:00:09.800885 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0703 22:00:09.901131 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0703 22:00:10.000621 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (99ms) I0703 22:00:10.101465 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (101ms) I0703 22:00:10.200662 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0703 22:00:10.300926 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0703 22:00:10.400114 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (99ms) I0703 22:00:10.500808 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0703 22:00:10.601022 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0703 22:00:10.700299 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (99ms) I0703 22:00:10.800341 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0703 22:00:10.900203 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0703 22:00:11.000851 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (101ms) I0703 22:00:11.032695 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:00:11.100920 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0703 22:00:11.200224 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (99ms) I0703 22:00:11.231458 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:00:11.300646 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0703 22:00:11.401263 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (101ms) I0703 22:00:11.500849 1 gather_workloads_info.go:387] No image sha256:88437b35173fe2e8a4d9f148ec17c754bee4df3f723b3a5676955eef4f363d31 (100ms) I0703 22:00:11.500881 1 tasks_processing.go:74] worker 1 stopped. E0703 22:00:11.500891 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0703 22:00:11.501176 1 recorder.go:75] Recording config/workload_info with fingerprint=c2f8440bb2e41329cffb5d64826b32b2476536dd19e529df565fa99e855041a7 I0703 22:00:11.501189 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.330825623s to process 1 records E0703 22:00:11.501211 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.331s with: function \"workload_info\" failed with an error" I0703 22:00:11.502310 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0703 22:00:11.502322 1 periodic.go:209] Running conditional gatherer I0703 22:00:11.507867 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0703 22:00:11.514423 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:51710->172.30.0.10:53: read: connection refused E0703 22:00:11.514653 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0703 22:00:11.514711 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0703 22:00:11.520382 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0703 22:00:11.520394 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520399 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520403 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520406 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520408 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520411 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520414 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520417 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:00:11.520420 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0703 22:00:11.520434 1 tasks_processing.go:45] number of workers: 3 I0703 22:00:11.520446 1 tasks_processing.go:69] worker 2 listening for tasks. I0703 22:00:11.520450 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0703 22:00:11.520460 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 22:00:11.520469 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 22:00:11.520478 1 tasks_processing.go:74] worker 1 stopped. I0703 22:00:11.520480 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0703 22:00:11.520485 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0703 22:00:11.520508 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0703 22:00:11.520519 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 796ns to process 1 records I0703 22:00:11.520600 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0703 22:00:11.520601 1 tasks_processing.go:74] worker 2 stopped. I0703 22:00:11.520629 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.376µs to process 1 records I0703 22:00:11.520736 1 tasks_processing.go:74] worker 0 stopped. I0703 22:00:11.520747 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 245.086µs to process 0 records I0703 22:00:11.520764 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:51710->172.30.0.10:53: read: connection refused I0703 22:00:11.520801 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0703 22:00:11.542798 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e6697b06ea73f0570f2275fcace9637535df555f9aba4ee2408db120a8e43c37 I0703 22:00:11.542946 1 diskrecorder.go:70] Writing 114 records to /var/lib/insights-operator/insights-2026-07-03-220011.tar.gz I0703 22:00:11.551155 1 diskrecorder.go:51] Wrote 114 records to disk in 8ms I0703 22:00:11.551183 1 periodic.go:278] Gathering cluster info every 2h0m0s I0703 22:00:11.551198 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0703 22:00:22.486726 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:01:19.603480 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="b80747186ba305bab1210400effbc541400f437df681c74a9a9addd426f30780") W0703 22:01:19.603515 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0703 22:01:19.603560 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="977d2809269cf1dc617e0fd45e5a0375849c7ee467f718896fc543960bfafb21") I0703 22:01:19.603594 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0703 22:01:19.603652 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="46c31ea1ca5ebeda518e09b05094f8a40bc9bb7b68edae7dddc22c02eef55954") I0703 22:01:19.603657 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0703 22:01:19.603657 1 periodic.go:170] Shutting down I0703 22:01:19.603671 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0703 22:01:19.603686 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0703 22:01:19.603692 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0703 22:01:19.603705 1 base_controller.go:181] Shutting down LoggingSyncer ...