W0320 11:05:19.204871 1 cmd.go:257] Using insecure, self-signed certificates I0320 11:05:19.717900 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:19.718235 1 observer_polling.go:159] Starting file observer I0320 11:05:20.146163 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0320 11:05:20.146354 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0320 11:05:20.146959 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0320 11:05:20.147460 1 secure_serving.go:57] Forcing use of http/1.1 only W0320 11:05:20.147485 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0320 11:05:20.147492 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0320 11:05:20.147499 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0320 11:05:20.147505 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0320 11:05:20.147510 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0320 11:05:20.147513 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0320 11:05:20.151290 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0320 11:05:20.151334 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"7989620d-3db1-40f9-b174-6273572ab932", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0320 11:05:20.152362 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0320 11:05:20.152372 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0320 11:05:20.152390 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0320 11:05:20.152394 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0320 11:05:20.152405 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 11:05:20.152409 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 11:05:20.152664 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3154362000/tls.crt::/tmp/serving-cert-3154362000/tls.key" I0320 11:05:20.153857 1 secure_serving.go:213] Serving securely on [::]:8443 I0320 11:05:20.153909 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0320 11:05:20.156669 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0320 11:05:20.156696 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 11:05:20.156813 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0320 11:05:20.161837 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:05:20.161855 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 11:05:20.166303 1 secretconfigobserver.go:119] support secret does not exist I0320 11:05:20.170980 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:05:20.175430 1 secretconfigobserver.go:119] support secret does not exist I0320 11:05:20.176822 1 recorder.go:176] Pruning old reports every 8h39m6s, max age is 288h0m0s I0320 11:05:20.181654 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0320 11:05:20.181661 1 periodic.go:216] Running clusterconfig gatherer I0320 11:05:20.181670 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0320 11:05:20.181697 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0320 11:05:20.181702 1 insightsreport.go:296] Starting report retriever I0320 11:05:20.181706 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0320 11:05:20.181709 1 tasks_processing.go:45] number of workers: 32 I0320 11:05:20.181731 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 11:05:20.181744 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:05:20.181747 1 tasks_processing.go:71] worker 2 working on cluster_apiserver task. I0320 11:05:20.181750 1 tasks_processing.go:69] worker 7 listening for tasks. I0320 11:05:20.181757 1 tasks_processing.go:69] worker 19 listening for tasks. I0320 11:05:20.181756 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:05:20.181762 1 tasks_processing.go:71] worker 7 working on validating_webhook_configurations task. I0320 11:05:20.181772 1 tasks_processing.go:69] worker 22 listening for tasks. I0320 11:05:20.181774 1 tasks_processing.go:69] worker 8 listening for tasks. I0320 11:05:20.181780 1 tasks_processing.go:69] worker 20 listening for tasks. I0320 11:05:20.181783 1 tasks_processing.go:69] worker 9 listening for tasks. I0320 11:05:20.181786 1 tasks_processing.go:69] worker 21 listening for tasks. I0320 11:05:20.181793 1 tasks_processing.go:69] worker 10 listening for tasks. I0320 11:05:20.181790 1 tasks_processing.go:69] worker 23 listening for tasks. I0320 11:05:20.181796 1 tasks_processing.go:69] worker 24 listening for tasks. I0320 11:05:20.181794 1 tasks_processing.go:69] worker 4 listening for tasks. I0320 11:05:20.181802 1 tasks_processing.go:69] worker 3 listening for tasks. I0320 11:05:20.181802 1 tasks_processing.go:69] worker 15 listening for tasks. I0320 11:05:20.181808 1 tasks_processing.go:69] worker 18 listening for tasks. I0320 11:05:20.181809 1 tasks_processing.go:69] worker 14 listening for tasks. I0320 11:05:20.181817 1 tasks_processing.go:69] worker 26 listening for tasks. I0320 11:05:20.181809 1 tasks_processing.go:69] worker 17 listening for tasks. I0320 11:05:20.181819 1 tasks_processing.go:69] worker 16 listening for tasks. I0320 11:05:20.181824 1 tasks_processing.go:69] worker 31 listening for tasks. I0320 11:05:20.181825 1 tasks_processing.go:69] worker 29 listening for tasks. I0320 11:05:20.181826 1 tasks_processing.go:69] worker 27 listening for tasks. I0320 11:05:20.181830 1 tasks_processing.go:69] worker 11 listening for tasks. I0320 11:05:20.181832 1 tasks_processing.go:69] worker 5 listening for tasks. I0320 11:05:20.181833 1 tasks_processing.go:69] worker 12 listening for tasks. I0320 11:05:20.181836 1 tasks_processing.go:69] worker 6 listening for tasks. I0320 11:05:20.181838 1 tasks_processing.go:69] worker 13 listening for tasks. I0320 11:05:20.181817 1 tasks_processing.go:69] worker 30 listening for tasks. I0320 11:05:20.181811 1 tasks_processing.go:69] worker 28 listening for tasks. I0320 11:05:20.181749 1 tasks_processing.go:71] worker 0 working on openstack_dataplanedeployments task. I0320 11:05:20.181860 1 tasks_processing.go:69] worker 25 listening for tasks. I0320 11:05:20.181911 1 tasks_processing.go:71] worker 1 working on pod_network_connectivity_checks task. I0320 11:05:20.181956 1 tasks_processing.go:71] worker 11 working on certificate_signing_requests task. I0320 11:05:20.181964 1 tasks_processing.go:71] worker 12 working on active_alerts task. W0320 11:05:20.181994 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:20.182012 1 tasks_processing.go:71] worker 12 working on ingress_certificates task. I0320 11:05:20.182072 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 41.481µs to process 0 records I0320 11:05:20.182134 1 tasks_processing.go:71] worker 30 working on openshift_machine_api_events task. I0320 11:05:20.181957 1 tasks_processing.go:71] worker 6 working on metrics task. W0320 11:05:20.182321 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:20.182347 1 tasks_processing.go:71] worker 6 working on machine_config_pools task. I0320 11:05:20.181916 1 tasks_processing.go:71] worker 25 working on node_features task. I0320 11:05:20.181921 1 tasks_processing.go:71] worker 3 working on machine_autoscalers task. I0320 11:05:20.181923 1 tasks_processing.go:71] worker 8 working on storage_cluster task. I0320 11:05:20.181925 1 tasks_processing.go:71] worker 15 working on storage_classes task. I0320 11:05:20.181927 1 tasks_processing.go:71] worker 20 working on machines task. I0320 11:05:20.181929 1 tasks_processing.go:71] worker 18 working on openstack_controlplanes task. I0320 11:05:20.181930 1 tasks_processing.go:71] worker 9 working on sap_config task. I0320 11:05:20.181932 1 tasks_processing.go:71] worker 14 working on sap_pods task. I0320 11:05:20.181934 1 tasks_processing.go:71] worker 21 working on mutating_webhook_configurations task. I0320 11:05:20.181935 1 tasks_processing.go:71] worker 26 working on networks task. I0320 11:05:20.181938 1 tasks_processing.go:71] worker 10 working on oauths task. I0320 11:05:20.181938 1 tasks_processing.go:71] worker 17 working on install_plans task. I0320 11:05:20.181942 1 tasks_processing.go:71] worker 23 working on qemu_kubevirt_launcher_logs task. I0320 11:05:20.181942 1 tasks_processing.go:71] worker 16 working on nodenetworkconfigurationpolicies task. I0320 11:05:20.181946 1 tasks_processing.go:71] worker 31 working on nodes task. I0320 11:05:20.181945 1 tasks_processing.go:71] worker 24 working on proxies task. I0320 11:05:20.181949 1 tasks_processing.go:71] worker 29 working on container_images task. I0320 11:05:20.181952 1 tasks_processing.go:71] worker 27 working on overlapping_namespace_uids task. I0320 11:05:20.181951 1 tasks_processing.go:71] worker 19 working on image_pruners task. I0320 11:05:20.181960 1 tasks_processing.go:71] worker 5 working on dvo_metrics task. I0320 11:05:20.182139 1 tasks_processing.go:71] worker 13 working on tsdb_status task. W0320 11:05:20.184235 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:20.181914 1 tasks_processing.go:71] worker 4 working on schedulers task. I0320 11:05:20.182147 1 tasks_processing.go:71] worker 28 working on operators_pods_and_events task. I0320 11:05:20.181919 1 tasks_processing.go:71] worker 22 working on monitoring_persistent_volumes task. I0320 11:05:20.182407 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 46.371µs to process 0 records I0320 11:05:20.184749 1 tasks_processing.go:71] worker 13 working on machine_sets task. I0320 11:05:20.185049 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 501.398µs to process 0 records I0320 11:05:20.185244 1 tasks_processing.go:71] worker 0 working on nodenetworkstates task. I0320 11:05:20.185408 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 3.390277ms to process 0 records I0320 11:05:20.186623 1 tasks_processing.go:71] worker 1 working on olm_operators task. E0320 11:05:20.186636 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0320 11:05:20.186794 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 4.658326ms to process 0 records I0320 11:05:20.186840 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0320 11:05:20.186862 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 11:05:20.186871 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 11:05:20.186876 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 11:05:20.186893 1 controller.go:531] The operator is still being initialized I0320 11:05:20.186903 1 controller.go:554] The operator is healthy I0320 11:05:20.187393 1 tasks_processing.go:71] worker 20 working on config_maps task. E0320 11:05:20.187404 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0320 11:05:20.187419 1 gather.go:177] gatherer "clusterconfig" function "machines" took 4.413036ms to process 0 records I0320 11:05:20.187578 1 tasks_processing.go:71] worker 2 working on image task. I0320 11:05:20.187752 1 recorder.go:75] Recording config/apiserver with fingerprint=178f6b1b55d33eb7342d9fc51934cafaad84bb9b446691d61cbba6ad310d5806 I0320 11:05:20.187768 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 5.815948ms to process 1 records E0320 11:05:20.187784 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0320 11:05:20.187793 1 tasks_processing.go:71] worker 25 working on machine_healthchecks task. I0320 11:05:20.187837 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 5.344415ms to process 0 records I0320 11:05:20.188465 1 tasks_processing.go:71] worker 6 working on openstack_dataplanenodesets task. I0320 11:05:20.188491 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 6.110369ms to process 0 records I0320 11:05:20.188503 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 5.818655ms to process 0 records I0320 11:05:20.188509 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 5.989028ms to process 0 records I0320 11:05:20.188542 1 tasks_processing.go:71] worker 8 working on node_logs task. I0320 11:05:20.188542 1 tasks_processing.go:71] worker 3 working on crds task. I0320 11:05:20.188769 1 tasks_processing.go:71] worker 14 working on clusterroles task. I0320 11:05:20.188795 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 5.521483ms to process 0 records I0320 11:05:20.189641 1 tasks_processing.go:71] worker 18 working on container_runtime_configs task. I0320 11:05:20.189656 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 6.522233ms to process 0 records I0320 11:05:20.191430 1 tasks_processing.go:71] worker 16 working on jaegers task. I0320 11:05:20.191446 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 7.788986ms to process 0 records I0320 11:05:20.191604 1 tasks_processing.go:71] worker 7 working on ceph_cluster task. I0320 11:05:20.191805 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=312796e286ab690c4d719380c1a1ffa94381bb776dbf26004449c876968729bf I0320 11:05:20.191885 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=51d9338ae46d750069a5d34f4845d0daf96d1941e9ea791cdbd4462ec3915b25 I0320 11:05:20.191904 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=f1fc841f14a09fc97c0a36416bdd76282bd24702ed7a81bd154e19576cecbfbe I0320 11:05:20.191927 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=beee4d24f4d9ab48d6aa2ff6add4014bd70c65d4ab00188aa0faacdceb9937af I0320 11:05:20.191950 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=cf86450e61d1b176afb550624db94dbe65367ea739f8ed796c61b7cf452f2450 I0320 11:05:20.191972 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=4b894ab992799af31313a51ecc8edbe9f02730f55a118ffb4d13b5390b771e15 I0320 11:05:20.191995 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=6c74b3abc7330e79e447278cb0644eef7afb9f51acf84ab96208f38df19305e5 I0320 11:05:20.192050 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=802bfab5b581b9eab83674905ef397d1df52355824e1fc2d4c39ebbc93dd4ef2 I0320 11:05:20.192078 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=32a3252a419f7f657e0fb8402535751ce69b7522d585443599fc9370b37786ec I0320 11:05:20.192101 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=4e6acfb5ecc7afca89b46245dfa24315b4b03a1fbb6465ab5f4bd8629a1d63d0 I0320 11:05:20.192123 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7c093fbbfb5a722a6342b41ec801f8a29d03e0584f12d2cbf8a7356bd34cef58 I0320 11:05:20.192129 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 9.827089ms to process 11 records I0320 11:05:20.192209 1 tasks_processing.go:71] worker 10 working on ingress task. I0320 11:05:20.192294 1 recorder.go:75] Recording config/oauth with fingerprint=f94c0012bd01bc496908e7386ac8db71cb0f4c3b628bddd3f4f01110fd50317a I0320 11:05:20.192302 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 8.498968ms to process 1 records I0320 11:05:20.192543 1 tasks_processing.go:71] worker 15 working on openshift_logging task. I0320 11:05:20.192637 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7a1ae141d0145ae1fbc877dabc65e5019ee7fa756134464eb75875ee6057c0b5 I0320 11:05:20.192651 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=429dc7df27c7ec0470a5e87b41877a50320b2fa983c0932bbc1dbb2585cc1358 I0320 11:05:20.192660 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 9.776199ms to process 2 records I0320 11:05:20.194546 1 tasks_processing.go:71] worker 11 working on aggregated_monitoring_cr_names task. I0320 11:05:20.194555 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 12.571727ms to process 0 records I0320 11:05:20.198705 1 tasks_processing.go:71] worker 13 working on openstack_version task. I0320 11:05:20.198721 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 13.943335ms to process 0 records I0320 11:05:20.198733 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 16.434569ms to process 0 records I0320 11:05:20.198795 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 9.137494ms to process 0 records I0320 11:05:20.198839 1 tasks_processing.go:71] worker 18 working on operators task. I0320 11:05:20.198841 1 tasks_processing.go:71] worker 30 working on service_accounts task. I0320 11:05:20.198984 1 tasks_processing.go:71] worker 9 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0320 11:05:20.199090 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 15.835207ms to process 0 records I0320 11:05:20.199131 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 14.117718ms to process 0 records I0320 11:05:20.199155 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 6.59721ms to process 0 records I0320 11:05:20.199170 1 tasks_processing.go:71] worker 15 working on sap_datahubs task. I0320 11:05:20.199175 1 tasks_processing.go:71] worker 22 working on feature_gates task. I0320 11:05:20.199598 1 tasks_processing.go:71] worker 16 working on infrastructures task. I0320 11:05:20.199610 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 8.153047ms to process 0 records I0320 11:05:20.199815 1 tasks_processing.go:71] worker 7 working on lokistack task. I0320 11:05:20.199838 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 8.197667ms to process 0 records I0320 11:05:20.199918 1 tasks_processing.go:71] worker 24 working on authentication task. I0320 11:05:20.200046 1 recorder.go:75] Recording config/proxy with fingerprint=641e0df56b46dbf3dd8b829128ac4e70fc80f0fbd34ee306dc4dadcd1af77e39 I0320 11:05:20.200064 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 16.067621ms to process 1 records I0320 11:05:20.200148 1 gather_logs.go:145] no pods in namespace were found I0320 11:05:20.200163 1 tasks_processing.go:71] worker 23 working on support_secret task. I0320 11:05:20.200176 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 16.563385ms to process 0 records I0320 11:05:20.200576 1 tasks_processing.go:71] worker 25 working on machine_configs task. E0320 11:05:20.200580 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0320 11:05:20.200590 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 12.769598ms to process 0 records I0320 11:05:20.200688 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=ce226550f00da1483e4deb92e5f2d13ebff8c7abfdbc1b98e89508702b7032e6 I0320 11:05:20.200724 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=215d6c099e581608a05e3d16ee14c2b358b21100d68524666a63f0b26b99fae9 I0320 11:05:20.200750 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=008b5b796de4205e7b296bc7dab405176a2fc81de4d28a65891dfeb0f00cf2f2 I0320 11:05:20.200760 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 17.29639ms to process 3 records I0320 11:05:20.200762 1 tasks_processing.go:71] worker 21 working on image_registries task. I0320 11:05:20.200863 1 tasks_processing.go:71] worker 6 working on pdbs task. I0320 11:05:20.200894 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 12.387973ms to process 0 records I0320 11:05:20.201912 1 tasks_processing.go:71] worker 0 working on cost_management_metrics_configs task. I0320 11:05:20.201915 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 16.625506ms to process 0 records I0320 11:05:20.202406 1 tasks_processing.go:71] worker 31 working on silenced_alerts task. W0320 11:05:20.202436 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:20.202710 1 recorder.go:75] Recording config/node/ip-10-0-0-215.ec2.internal with fingerprint=9e210af3d8800c8952e79af4e4553e4b96a509c12dabbd2201418b1a8130bc0e I0320 11:05:20.202769 1 recorder.go:75] Recording config/node/ip-10-0-1-143.ec2.internal with fingerprint=0e0cf9154655d6650a5bbb04b9ebc834f6826f690d6c6e1a0d9d2c385e42b3f7 I0320 11:05:20.202839 1 recorder.go:75] Recording config/node/ip-10-0-2-149.ec2.internal with fingerprint=ee4c24473ef937bac595c2fe426cc2cc5ef76d2219a0a9882dbab238fe2927fc I0320 11:05:20.202850 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 18.658674ms to process 3 records I0320 11:05:20.202856 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 25.24µs to process 0 records I0320 11:05:20.202871 1 tasks_processing.go:71] worker 31 working on version task. I0320 11:05:20.203568 1 tasks_processing.go:74] worker 26 stopped. I0320 11:05:20.203768 1 recorder.go:75] Recording config/network with fingerprint=0c1da67edd076d3c27d02b421a6431a5a7db5f9405f517070f24ff7274e38d3d I0320 11:05:20.203784 1 gather.go:177] gatherer "clusterconfig" function "networks" took 20.176075ms to process 1 records I0320 11:05:20.204087 1 tasks_processing.go:74] worker 13 stopped. I0320 11:05:20.204100 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 5.364143ms to process 0 records I0320 11:05:20.205665 1 tasks_processing.go:74] worker 10 stopped. I0320 11:05:20.205811 1 recorder.go:75] Recording config/ingress with fingerprint=e65bc8624e506642e21b4c316168ec606c5425da978c2067763e81ce99a5c112 I0320 11:05:20.205824 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 13.436779ms to process 1 records I0320 11:05:20.208895 1 tasks_processing.go:74] worker 2 stopped. I0320 11:05:20.209138 1 recorder.go:75] Recording config/image with fingerprint=5e00a2237c65447d053825e02995b14f3a99cecfebe1433c6fd54ad5b4f57d13 I0320 11:05:20.209204 1 gather.go:177] gatherer "clusterconfig" function "image" took 21.302487ms to process 1 records I0320 11:05:20.210819 1 tasks_processing.go:74] worker 19 stopped. I0320 11:05:20.211171 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=91193f674ec69b6cbcf6ef422b0779b97920f5f3dce62a941773300f9e9e48b3 I0320 11:05:20.211189 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 26.749272ms to process 1 records W0320 11:05:20.213168 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:05:20.216612 1 tasks_processing.go:74] worker 15 stopped. I0320 11:05:20.216628 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 17.431823ms to process 0 records I0320 11:05:20.216754 1 tasks_processing.go:74] worker 7 stopped. I0320 11:05:20.216802 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 16.927321ms to process 0 records I0320 11:05:20.216973 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0320 11:05:20.217072 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0320 11:05:20.217105 1 operator.go:328] started I0320 11:05:20.217125 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0320 11:05:20.217320 1 tasks_processing.go:74] worker 16 stopped. I0320 11:05:20.218243 1 recorder.go:75] Recording config/infrastructure with fingerprint=c532cdcc6ca2d1f415c308183a6756d75b84a0df60293d767297f9b62e5da0a4 I0320 11:05:20.218262 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 17.704762ms to process 1 records I0320 11:05:20.218287 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0320 11:05:20.218298 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 33.646255ms to process 1 records E0320 11:05:20.218311 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0320 11:05:20.218317 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 17.620665ms to process 0 records I0320 11:05:20.218335 1 tasks_processing.go:74] worker 27 stopped. I0320 11:05:20.218337 1 tasks_processing.go:74] worker 23 stopped. I0320 11:05:20.218409 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=da046e7aa069ddf756a999211d9cf6bde2f0eb6315eff32794eface17f97dbf3 I0320 11:05:20.218410 1 tasks_processing.go:74] worker 4 stopped. I0320 11:05:20.218433 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 33.567418ms to process 1 records I0320 11:05:20.218439 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 16.16949ms to process 0 records I0320 11:05:20.218459 1 tasks_processing.go:74] worker 0 stopped. I0320 11:05:20.218535 1 tasks_processing.go:74] worker 24 stopped. I0320 11:05:20.218604 1 recorder.go:75] Recording config/authentication with fingerprint=35d58f08f80e044a3097b73c7c1b3e784ceaedfbb00e8e023766c892f0e39ad1 I0320 11:05:20.218615 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 18.229359ms to process 1 records I0320 11:05:20.218695 1 tasks_processing.go:74] worker 14 stopped. I0320 11:05:20.218755 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=4e3a3e0e5716af4dd997ea30a68211c747f1b70b6e3f13a67f991f7853f9efe6 I0320 11:05:20.218834 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2f5fd4e312768edd521d9bc6db0752b38f7f44c670b2261bd9b1dec0ad59b51d I0320 11:05:20.218841 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 29.548461ms to process 2 records I0320 11:05:20.220545 1 tasks_processing.go:74] worker 21 stopped. I0320 11:05:20.221017 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f623563d95309780e8d44bdda3744f918a9058b4653b77f2434f83bc770f9f01 I0320 11:05:20.221054 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 19.767007ms to process 1 records I0320 11:05:20.221637 1 tasks_processing.go:74] worker 22 stopped. I0320 11:05:20.221767 1 recorder.go:75] Recording config/featuregate with fingerprint=33aa58605287662929abb35b2e27e5da5a3ecb5e86e293db9b39cdae80e5bb7a I0320 11:05:20.221793 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 22.44784ms to process 1 records I0320 11:05:20.222597 1 tasks_processing.go:74] worker 1 stopped. I0320 11:05:20.222661 1 recorder.go:75] Recording config/olm_operators with fingerprint=cbbf46ff060a45ab9e6baf433e5e5fcbeda37ef7dccd60eb8767bdec4af04f01 I0320 11:05:20.222677 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 35.935272ms to process 1 records I0320 11:05:20.222753 1 tasks_processing.go:74] worker 6 stopped. I0320 11:05:20.222863 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=2e1f08ad8f501a6bc01b151b619a9ae135d0f5087d6de054cc78a795dd60d141 I0320 11:05:20.222900 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a57ad70801327300961c0739052a34d28b52229211b7cd90130dbf60ff2ff5f9 I0320 11:05:20.222934 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=2039ca3da327906fefaae92a2bd7c03bf285b6548808c2a7e41c54149fcb6edb I0320 11:05:20.222947 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 21.870796ms to process 3 records I0320 11:05:20.227820 1 tasks_processing.go:74] worker 25 stopped. E0320 11:05:20.227833 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0320 11:05:20.227853 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0320 11:05:20.227867 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 27.228928ms to process 1 records I0320 11:05:20.227951 1 tasks_processing.go:74] worker 3 stopped. I0320 11:05:20.228474 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=44fe7fc8e5d67074abe8ba7c246d805c2aa69a6f272f3e0a4a12b9bbc3fdbdd3 I0320 11:05:20.229239 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=a576230ce1fe87a26a38e9ba82ccd5bd2e899e55adaf6c3d512b9619729af20f I0320 11:05:20.229269 1 gather.go:177] gatherer "clusterconfig" function "crds" took 39.22863ms to process 2 records I0320 11:05:20.236348 1 tasks_processing.go:74] worker 29 stopped. I0320 11:05:20.237644 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-kchcs with fingerprint=24c55ee5d429f61fa783196a8f81d071b921e6a7f358ffcb5b5b1462b119a5d6 I0320 11:05:20.237866 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-xcjt2 with fingerprint=3754941785a030717729d451110a88e75ba97137f4dcf68d3f4abfd3e2fff447 I0320 11:05:20.237911 1 recorder.go:75] Recording config/running_containers with fingerprint=135b018e5988e5699a350828f75eaf475e8c82035cc13a2f7a70ceadd60a9e90 I0320 11:05:20.237920 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 52.449862ms to process 3 records I0320 11:05:20.238830 1 prometheus_rules.go:88] Prometheus rules successfully created I0320 11:05:20.239001 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 11:05:20.239013 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 11:05:20.239016 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 11:05:20.239032 1 controller.go:254] Source scaController *sca.Controller is not ready I0320 11:05:20.239035 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0320 11:05:20.239052 1 controller.go:531] The operator is still being initialized I0320 11:05:20.239057 1 controller.go:554] The operator is healthy I0320 11:05:20.239071 1 tasks_processing.go:74] worker 11 stopped. I0320 11:05:20.239085 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 44.50662ms to process 0 records E0320 11:05:20.247607 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27fade2ce1-cbab-4de1-8e87-9348a8e820ac%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.12:38787->172.30.0.10:53: read: connection refused I0320 11:05:20.247621 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27fade2ce1-cbab-4de1-8e87-9348a8e820ac%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.12:38787->172.30.0.10:53: read: connection refused I0320 11:05:20.252420 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0320 11:05:20.252429 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 11:05:20.252445 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 11:05:20.256864 1 base_controller.go:82] Caches are synced for ConfigController I0320 11:05:20.256880 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0320 11:05:20.263702 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:05:20.264235 1 tasks_processing.go:74] worker 20 stopped. E0320 11:05:20.264251 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0320 11:05:20.264261 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0320 11:05:20.264264 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0320 11:05:20.264273 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=c8a9805405aae7d04b7f03c09c51097942070877009b6daebd5573ef8127446e I0320 11:05:20.264306 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0320 11:05:20.264313 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0320 11:05:20.264317 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=3b70b9787487b548204d806ebebf90b0c2d460145e34cb33c0d8d3a7a011d907 I0320 11:05:20.264322 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0320 11:05:20.264377 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0320 11:05:20.264387 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0320 11:05:20.264393 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 76.819595ms to process 7 records I0320 11:05:20.294753 1 tasks_processing.go:74] worker 8 stopped. I0320 11:05:20.294766 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 106.189861ms to process 0 records I0320 11:05:20.295693 1 tasks_processing.go:74] worker 31 stopped. I0320 11:05:20.295917 1 recorder.go:75] Recording config/version with fingerprint=aea2091e91cd81c6d18b7d554d0516b3476b3bc41a493e3e8da297b43915d995 I0320 11:05:20.295930 1 recorder.go:75] Recording config/id with fingerprint=49fb132acd73404ed15552b9037a263d06c14a7da256de1a9c53330b03b706ce I0320 11:05:20.295936 1 gather.go:177] gatherer "clusterconfig" function "version" took 92.809283ms to process 2 records I0320 11:05:20.305074 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0320 11:05:20.312767 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.12:43607->172.30.0.10:53: read: connection refused I0320 11:05:20.312783 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.12:43607->172.30.0.10:53: read: connection refused I0320 11:05:20.317407 1 base_controller.go:82] Caches are synced for LoggingSyncer I0320 11:05:20.317419 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0320 11:05:20.317445 1 tasks_processing.go:74] worker 9 stopped. I0320 11:05:20.317453 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 118.447156ms to process 0 records I0320 11:05:20.332203 1 tasks_processing.go:74] worker 12 stopped. E0320 11:05:20.332217 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0320 11:05:20.332224 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p5dfcheert4iiujdq8kqp8439v2eqd2-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p5dfcheert4iiujdq8kqp8439v2eqd2-primary-cert-bundle-secret" not found I0320 11:05:20.332283 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=56f7e48a603448f9833f0e15b3cad12c1c9c6bf1187c86a5c9cededbef8eb708 I0320 11:05:20.332296 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 150.18344ms to process 1 records I0320 11:05:20.615430 1 gather_cluster_operator_pods_and_events.go:121] Found 18 pods with 21 containers I0320 11:05:20.615442 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1198372 bytes I0320 11:05:20.615625 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-8tt59 pod in namespace openshift-dns (previous: false). I0320 11:05:20.842552 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-8tt59 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-8tt59\" is waiting to start: ContainerCreating" I0320 11:05:20.842570 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-8tt59\" is waiting to start: ContainerCreating" I0320 11:05:20.842577 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-8tt59 pod in namespace openshift-dns (previous: false). I0320 11:05:21.023561 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-8tt59 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-8tt59\" is waiting to start: ContainerCreating" I0320 11:05:21.023576 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-8tt59\" is waiting to start: ContainerCreating" I0320 11:05:21.023590 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-kz7qk pod in namespace openshift-dns (previous: false). W0320 11:05:21.211244 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:05:21.244854 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kz7qk pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-kz7qk\" is waiting to start: ContainerCreating" I0320 11:05:21.244870 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-kz7qk\" is waiting to start: ContainerCreating" I0320 11:05:21.244879 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-kz7qk pod in namespace openshift-dns (previous: false). I0320 11:05:21.417253 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kz7qk pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-kz7qk\" is waiting to start: ContainerCreating" I0320 11:05:21.417267 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-kz7qk\" is waiting to start: ContainerCreating" I0320 11:05:21.417293 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-s9kmw pod in namespace openshift-dns (previous: false). I0320 11:05:21.619987 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-s9kmw pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-s9kmw\" is waiting to start: ContainerCreating" I0320 11:05:21.620004 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-s9kmw\" is waiting to start: ContainerCreating" I0320 11:05:21.620012 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-s9kmw pod in namespace openshift-dns (previous: false). I0320 11:05:21.817139 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-s9kmw pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-s9kmw\" is waiting to start: ContainerCreating" I0320 11:05:21.817166 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-s9kmw\" is waiting to start: ContainerCreating" I0320 11:05:21.817174 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-79hqc pod in namespace openshift-dns (previous: false). I0320 11:05:21.849805 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0320 11:05:22.016500 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:22.016517 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-jbnf4 pod in namespace openshift-dns (previous: false). W0320 11:05:22.210393 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:05:22.217254 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:22.217268 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-zn8ms pod in namespace openshift-dns (previous: false). I0320 11:05:22.251334 1 tasks_processing.go:74] worker 18 stopped. I0320 11:05:22.251377 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=d9b7bb395b3049b3e8f4b3d0690efa69b66c6abc48f897a703a2a9a3645cd54e I0320 11:05:22.251415 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=2d0177d3985a8b93d98cc7c60bd29a86c65db5ae6ee81e4c53adf3ab78115c10 I0320 11:05:22.251446 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0320 11:05:22.251486 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=26831c76b3b7d1c312c74ff40aa3f9f6b2eae7a35c173d65ec12a2f14f993e6b I0320 11:05:22.251503 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0320 11:05:22.251525 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=7178cea8dac85d93c6261f61483a33da700033b6d6d4c02c554842a4b2d9e554 I0320 11:05:22.251556 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=ad9df726660ba215c883d4f27586c6f3b57539f9eb54ee64f374cee9bceb4009 I0320 11:05:22.251580 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=006e33ef56d3e6ec7e97d357d117c691ebea5bf03534421d4c11907ad2e951eb I0320 11:05:22.251610 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=3df7cdfb312348b3f53723a0ada3f92ad80cedee725d4199773f5c5cd6be99e5 I0320 11:05:22.251620 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0320 11:05:22.251638 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=e70ca4dfcd2e42fdb04cde0c350c0992d760beb126d394708041e0d53f3d618d I0320 11:05:22.251648 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0320 11:05:22.251664 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=6a48fa922188a02e65408adc521452c14beb71cb832b79d9826cfa8d91a9984f I0320 11:05:22.251685 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0320 11:05:22.251702 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=d0af31a0fa2c2c985fee5ff5998a7227b10fa6a0fd7d4d102098e000594a48a2 I0320 11:05:22.251710 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0320 11:05:22.251722 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=e42948eccad1d5017f5be438d100b2c817ad19e3059d330da8c33fb8abdffe5e I0320 11:05:22.251735 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0320 11:05:22.251751 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=53f91df9af0a3587516a1de2cb8f3fe7e113ac1458007d991ce4eb64123f4960 I0320 11:05:22.251872 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=026e2db9f27e754e7f65f3b7b076c7cd2cc70bac4b44556dbe9dc376709e7c55 I0320 11:05:22.251881 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0320 11:05:22.251888 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0320 11:05:22.251909 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0320 11:05:22.251931 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=4408a66b2245cf539ca0ab6119e58ad2630e63f4622cd1b591797c26f0b7d1d2 I0320 11:05:22.251953 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=61f987c8849afc25a0e548dc027a9d32de81983e0ea5d76997d4d18f6dd3f38f I0320 11:05:22.251963 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0320 11:05:22.251978 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=ec56a33da6a4dfe9939875e788b95c698df4f0c3060e05ae76d3131dc9561e2b I0320 11:05:22.251985 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0320 11:05:22.251998 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=c478c268aab5da5f47a2a314506124f1505517c00fee62a393e5b02037c25b22 I0320 11:05:22.252012 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=5b44641ec83a8c4e22951a3c9c19c2921e93639a3c36af6dda359c811633e85c I0320 11:05:22.252046 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b59eb8a311127c67fcd5effb4c9e1f3e50c0f23d172087ada473bedba328e63d I0320 11:05:22.252067 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=e345ef661413cd263cb2089a0886830dabb499cf1e064bdff403176de5bea7d3 I0320 11:05:22.252082 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=b9ad566250ec63eab30a7416692b9b4f3ab6fc18644b1a3e22eb4c5652440f5b I0320 11:05:22.252104 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b3f11f46505b5772d1b72f4cd8e5d0790e2833c25129c7e5faee0866f5db993f I0320 11:05:22.252121 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0320 11:05:22.252127 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0320 11:05:22.252134 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.052476812s to process 36 records I0320 11:05:22.417148 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:22.417192 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7c88c469c5-kkz7p pod in namespace openshift-image-registry (previous: false). I0320 11:05:22.616152 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-7c88c469c5-kkz7p pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-7c88c469c5-kkz7p\" is waiting to start: ContainerCreating" I0320 11:05:22.616166 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-7c88c469c5-kkz7p\" is waiting to start: ContainerCreating" I0320 11:05:22.616203 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-9f78cb956-shtq2 pod in namespace openshift-image-registry (previous: false). I0320 11:05:22.817277 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-9f78cb956-shtq2 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-9f78cb956-shtq2\" is waiting to start: ContainerCreating" I0320 11:05:22.817290 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-9f78cb956-shtq2\" is waiting to start: ContainerCreating" I0320 11:05:22.817345 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-9f78cb956-w87j5 pod in namespace openshift-image-registry (previous: false). I0320 11:05:23.017859 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-9f78cb956-w87j5 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-9f78cb956-w87j5\" is waiting to start: ContainerCreating" I0320 11:05:23.017872 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-9f78cb956-w87j5\" is waiting to start: ContainerCreating" I0320 11:05:23.017881 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-czkzd pod in namespace openshift-image-registry (previous: false). W0320 11:05:23.210554 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:05:23.217634 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:23.217648 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-jbgwp pod in namespace openshift-image-registry (previous: false). I0320 11:05:23.417406 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:23.417419 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-m247k pod in namespace openshift-image-registry (previous: false). I0320 11:05:23.617970 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0320 11:05:23.617988 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6956544bb9-npdqf pod in namespace openshift-ingress (previous: false). I0320 11:05:23.817765 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6956544bb9-npdqf pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6956544bb9-npdqf\" is waiting to start: ContainerCreating" I0320 11:05:23.817781 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6956544bb9-npdqf\" is waiting to start: ContainerCreating" I0320 11:05:23.817792 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7459997687-62j9b pod in namespace openshift-ingress (previous: false). I0320 11:05:24.016854 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7459997687-62j9b pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7459997687-62j9b\" is waiting to start: ContainerCreating" I0320 11:05:24.016870 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7459997687-62j9b\" is waiting to start: ContainerCreating" I0320 11:05:24.016881 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7459997687-gswkv pod in namespace openshift-ingress (previous: false). W0320 11:05:24.210674 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:05:24.217737 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7459997687-gswkv pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7459997687-gswkv\" is waiting to start: ContainerCreating" I0320 11:05:24.217753 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7459997687-gswkv\" is waiting to start: ContainerCreating" I0320 11:05:24.217766 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-2lq8m pod in namespace openshift-ingress-canary (previous: false). I0320 11:05:24.417593 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-h8m8g pod in namespace openshift-ingress-canary (previous: false). I0320 11:05:24.617166 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-kq98z pod in namespace openshift-ingress-canary (previous: false). I0320 11:05:24.817682 1 tasks_processing.go:74] worker 28 stopped. I0320 11:05:24.817819 1 recorder.go:75] Recording events/openshift-dns with fingerprint=4b6d39a64ecaf6da9bd1a15655a891a3f3803f74506cce8240e24a613e7d1ae7 I0320 11:05:24.817951 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=cb869400ec53d93bf2abce71fcad5ce90e87141a95dd03d3398d46b64dc623c0 I0320 11:05:24.817980 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=2a832fa47d2d7d830ab73917f351df42b57b089886f354a9ce8c79d625ab8463 I0320 11:05:24.818041 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=73a74c71b3c9ac6a9a8031c01f75d829b2e3cb81d2371b927ce419ef19f26e5c I0320 11:05:24.818079 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=b8305ee032547b8d57301d0458e41a897020a12898cadeec3d3ce381cc86f220 I0320 11:05:24.818222 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-s9kmw with fingerprint=7702a351db61119480d26008b4f1ad2d8a0ab3dd6fd5d8887cef8b5de79de980 I0320 11:05:24.818356 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-7c88c469c5-kkz7p with fingerprint=1480c67ec4fcce589d24ad1fdcd13bb7211c65af0e4c9ffcd7c62393a963fb44 I0320 11:05:24.818445 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-9f78cb956-shtq2 with fingerprint=665edc1748259092d73a01931f4cd937e83f8bea0c08bf6f33f369bc6514e4ba I0320 11:05:24.818538 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-9f78cb956-w87j5 with fingerprint=ce27a91dd85ff5d3bb657a3d7d5ea7f71823d8783c09ef51903c97fb3893f0ea I0320 11:05:24.818545 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-2lq8m/serve-healthcheck-canary_current.log with fingerprint=e30f89675da1d5688f8468341c26b05305b0e9702a57c6dba4578e0639c73e2c I0320 11:05:24.818549 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-h8m8g/serve-healthcheck-canary_current.log with fingerprint=2f00c269f068abd8c90ca7f4649f8b4eb8ffadb6e557b34a6b17cddab74c8499 I0320 11:05:24.818554 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-kq98z/serve-healthcheck-canary_current.log with fingerprint=e3fbd4cc52e8e9657b11aecee436cd608a943054d42a478da4ccd28236e97552 I0320 11:05:24.818561 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.633212774s to process 12 records W0320 11:05:25.207353 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 11:05:25.207376 1 tasks_processing.go:74] worker 5 stopped. E0320 11:05:25.207399 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 11:05:25.207419 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0320 11:05:25.207434 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0320 11:05:25.207445 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.023280881s to process 1 records I0320 11:05:32.617489 1 tasks_processing.go:74] worker 17 stopped. I0320 11:05:32.617529 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0320 11:05:32.617543 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.433965443s to process 1 records I0320 11:05:33.172241 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:05:33.405611 1 tasks_processing.go:74] worker 30 stopped. I0320 11:05:33.405981 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=4ce39b12c057291c1cbb2425e1f57ef4a8cb96c61f6227345caf83b89c351d65 I0320 11:05:33.406007 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.206632632s to process 1 records E0320 11:05:33.406148 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.224s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0320 11:05:33.407268 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0320 11:05:33.407288 1 periodic.go:216] Running workloads gatherer I0320 11:05:33.407311 1 tasks_processing.go:45] number of workers: 2 I0320 11:05:33.407325 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:05:33.407332 1 tasks_processing.go:71] worker 1 working on workload_info task. I0320 11:05:33.407343 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:05:33.407414 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0320 11:05:33.433359 1 tasks_processing.go:74] worker 0 stopped. I0320 11:05:33.433378 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.901284ms to process 0 records I0320 11:05:33.445993 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0320 11:05:33.454096 1 gather_workloads_info.go:387] No image sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575 (9ms) I0320 11:05:33.469272 1 gather_workloads_info.go:387] No image sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319 (15ms) I0320 11:05:33.477730 1 gather_workloads_info.go:387] No image sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6 (8ms) I0320 11:05:33.486054 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (8ms) I0320 11:05:33.495158 1 gather_workloads_info.go:387] No image sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7 (9ms) I0320 11:05:33.503965 1 gather_workloads_info.go:387] No image sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a (9ms) I0320 11:05:33.512223 1 gather_workloads_info.go:387] No image sha256:0b2cc6aadb1abc1d35a55da0c9884eb9426d7ff7a97174daeb09bb1e1a23cfae (8ms) I0320 11:05:33.520141 1 gather_workloads_info.go:387] No image sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211 (8ms) I0320 11:05:33.527958 1 gather_workloads_info.go:387] No image sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574 (8ms) I0320 11:05:33.538598 1 gather_workloads_info.go:387] No image sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329 (11ms) I0320 11:05:33.553951 1 gather_workloads_info.go:387] No image sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648 (15ms) I0320 11:05:33.654404 1 gather_workloads_info.go:387] No image sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044 (100ms) I0320 11:05:33.755041 1 gather_workloads_info.go:387] No image sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c (101ms) I0320 11:05:33.854368 1 gather_workloads_info.go:387] No image sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132 (99ms) I0320 11:05:33.955630 1 gather_workloads_info.go:387] No image sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd (101ms) I0320 11:05:34.055290 1 gather_workloads_info.go:387] No image sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df (100ms) I0320 11:05:34.154146 1 gather_workloads_info.go:387] No image sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809 (99ms) I0320 11:05:34.254646 1 gather_workloads_info.go:387] No image sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e (100ms) I0320 11:05:34.354494 1 gather_workloads_info.go:387] No image sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a (100ms) I0320 11:05:34.454902 1 gather_workloads_info.go:387] No image sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae (100ms) I0320 11:05:34.555349 1 gather_workloads_info.go:387] No image sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc (100ms) I0320 11:05:34.655409 1 gather_workloads_info.go:387] No image sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4 (100ms) I0320 11:05:34.754352 1 gather_workloads_info.go:387] No image sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740 (99ms) I0320 11:05:34.854784 1 gather_workloads_info.go:387] No image sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958 (100ms) I0320 11:05:34.954794 1 gather_workloads_info.go:387] No image sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd (100ms) I0320 11:05:35.055135 1 gather_workloads_info.go:387] No image sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004 (100ms) I0320 11:05:35.154162 1 gather_workloads_info.go:387] No image sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498 (99ms) I0320 11:05:35.254592 1 gather_workloads_info.go:387] No image sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f (100ms) I0320 11:05:35.355582 1 gather_workloads_info.go:387] No image sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6 (101ms) I0320 11:05:35.454955 1 gather_workloads_info.go:387] No image sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb (99ms) I0320 11:05:35.554942 1 gather_workloads_info.go:387] No image sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f (100ms) I0320 11:05:35.654299 1 gather_workloads_info.go:387] No image sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635 (99ms) I0320 11:05:35.654333 1 tasks_processing.go:74] worker 1 stopped. E0320 11:05:35.654343 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0320 11:05:35.654572 1 recorder.go:75] Recording config/workload_info with fingerprint=2706ef30ce25b9137e34a48c5ca3deade34b2274359782d8eb66138a07ccd684 I0320 11:05:35.654587 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.24699351s to process 1 records E0320 11:05:35.654613 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.247s with: function \"workload_info\" failed with an error" I0320 11:05:35.655712 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0320 11:05:35.655724 1 periodic.go:216] Running conditional gatherer I0320 11:05:35.661524 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules I0320 11:05:35.668113 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.12:39785->172.30.0.10:53: read: connection refused E0320 11:05:35.668403 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0320 11:05:35.668462 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0320 11:05:35.673801 1 conditional_gatherer.go:392] cluster version is '4.21.5' E0320 11:05:35.673816 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673822 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673826 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673829 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673832 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673834 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673837 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673840 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:05:35.673843 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0320 11:05:35.673856 1 tasks_processing.go:45] number of workers: 3 I0320 11:05:35.673874 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 11:05:35.673882 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0320 11:05:35.673886 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:05:35.673897 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0320 11:05:35.673899 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:05:35.673906 1 tasks_processing.go:74] worker 1 stopped. I0320 11:05:35.673913 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0320 11:05:35.673969 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0320 11:05:35.673982 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.386µs to process 1 records I0320 11:05:35.674013 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0320 11:05:35.674040 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.409µs to process 1 records I0320 11:05:35.674046 1 tasks_processing.go:74] worker 0 stopped. I0320 11:05:35.674235 1 tasks_processing.go:74] worker 2 stopped. I0320 11:05:35.674248 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 308.444µs to process 0 records I0320 11:05:35.674267 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.12:39785->172.30.0.10:53: read: connection refused I0320 11:05:35.674286 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0320 11:05:35.699043 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=a3eb212ad4c0c88833f2c68e37e53cb33847f77cce55dd8f9eab20056ec9b249 I0320 11:05:35.699179 1 diskrecorder.go:70] Writing 109 records to /var/lib/insights-operator/insights-2026-03-20-110535.tar.gz I0320 11:05:35.705846 1 diskrecorder.go:51] Wrote 109 records to disk in 6ms I0320 11:05:35.705889 1 periodic.go:285] Gathering cluster info every 2h0m0s I0320 11:05:35.705905 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0320 11:05:36.183926 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:05:36.385334 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:05:49.437735 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:06:34.719377 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="fbfc1912987a1d5f59cbcb2e1f4e290d2b76e572403c7b455f8a6a028a4ac5be") W0320 11:06:34.719412 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0320 11:06:34.719462 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector