W0405 11:05:39.101801 1 cmd.go:257] Using insecure, self-signed certificates I0405 11:05:39.809331 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:39.809665 1 observer_polling.go:159] Starting file observer I0405 11:05:41.007907 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0405 11:05:41.008101 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0405 11:05:41.008540 1 secure_serving.go:57] Forcing use of http/1.1 only W0405 11:05:41.008559 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0405 11:05:41.008564 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0405 11:05:41.008568 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0405 11:05:41.008571 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0405 11:05:41.008576 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0405 11:05:41.008578 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0405 11:05:41.008710 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0405 11:05:41.011486 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0405 11:05:41.011524 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"5b4148c0-5f4d-4645-8f10-52d0868bd039", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0405 11:05:41.016966 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0405 11:05:41.016981 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0405 11:05:41.016989 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0405 11:05:41.017005 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0405 11:05:41.016995 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0405 11:05:41.016987 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0405 11:05:41.017193 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3938036512/tls.crt::/tmp/serving-cert-3938036512/tls.key" I0405 11:05:41.017404 1 secure_serving.go:213] Serving securely on [::]:8443 I0405 11:05:41.017429 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0405 11:05:41.024329 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0405 11:05:41.024349 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0405 11:05:41.024437 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0405 11:05:41.028453 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0405 11:05:41.028468 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0405 11:05:41.032938 1 secretconfigobserver.go:119] support secret does not exist I0405 11:05:41.038861 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0405 11:05:41.043078 1 secretconfigobserver.go:119] support secret does not exist I0405 11:05:41.044330 1 recorder.go:161] Pruning old reports every 7h12m24s, max age is 288h0m0s I0405 11:05:41.048506 1 periodic.go:209] Running clusterconfig gatherer I0405 11:05:41.048520 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0405 11:05:41.048536 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0405 11:05:41.048540 1 tasks_processing.go:45] number of workers: 64 I0405 11:05:41.048560 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0405 11:05:41.048569 1 insightsreport.go:296] Starting report retriever I0405 11:05:41.048575 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0405 11:05:41.048585 1 tasks_processing.go:69] worker 63 listening for tasks. I0405 11:05:41.048593 1 tasks_processing.go:71] worker 63 working on machine_healthchecks task. I0405 11:05:41.048594 1 tasks_processing.go:69] worker 34 listening for tasks. I0405 11:05:41.048591 1 tasks_processing.go:69] worker 23 listening for tasks. I0405 11:05:41.048605 1 tasks_processing.go:69] worker 38 listening for tasks. I0405 11:05:41.048606 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 11:05:41.048601 1 tasks_processing.go:69] worker 33 listening for tasks. I0405 11:05:41.048617 1 tasks_processing.go:69] worker 37 listening for tasks. I0405 11:05:41.048608 1 tasks_processing.go:69] worker 39 listening for tasks. I0405 11:05:41.048615 1 tasks_processing.go:69] worker 40 listening for tasks. I0405 11:05:41.048625 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 11:05:41.048624 1 tasks_processing.go:69] worker 41 listening for tasks. I0405 11:05:41.048630 1 tasks_processing.go:69] worker 31 listening for tasks. I0405 11:05:41.048632 1 tasks_processing.go:69] worker 42 listening for tasks. I0405 11:05:41.048624 1 tasks_processing.go:69] worker 28 listening for tasks. I0405 11:05:41.048632 1 tasks_processing.go:69] worker 36 listening for tasks. I0405 11:05:41.048643 1 tasks_processing.go:69] worker 35 listening for tasks. I0405 11:05:41.048642 1 tasks_processing.go:69] worker 26 listening for tasks. I0405 11:05:41.048614 1 tasks_processing.go:69] worker 29 listening for tasks. I0405 11:05:41.048657 1 tasks_processing.go:69] worker 19 listening for tasks. I0405 11:05:41.048660 1 tasks_processing.go:69] worker 4 listening for tasks. I0405 11:05:41.048645 1 tasks_processing.go:69] worker 24 listening for tasks. I0405 11:05:41.048656 1 tasks_processing.go:69] worker 9 listening for tasks. I0405 11:05:41.048667 1 tasks_processing.go:69] worker 6 listening for tasks. I0405 11:05:41.048663 1 tasks_processing.go:69] worker 27 listening for tasks. I0405 11:05:41.048673 1 tasks_processing.go:69] worker 11 listening for tasks. I0405 11:05:41.048676 1 tasks_processing.go:69] worker 61 listening for tasks. I0405 11:05:41.048675 1 tasks_processing.go:69] worker 21 listening for tasks. I0405 11:05:41.048679 1 tasks_processing.go:69] worker 49 listening for tasks. I0405 11:05:41.048676 1 tasks_processing.go:69] worker 5 listening for tasks. I0405 11:05:41.048684 1 tasks_processing.go:69] worker 59 listening for tasks. I0405 11:05:41.048686 1 tasks_processing.go:69] worker 50 listening for tasks. I0405 11:05:41.048688 1 tasks_processing.go:69] worker 48 listening for tasks. I0405 11:05:41.048683 1 tasks_processing.go:69] worker 58 listening for tasks. I0405 11:05:41.048694 1 tasks_processing.go:69] worker 52 listening for tasks. I0405 11:05:41.048696 1 tasks_processing.go:69] worker 54 listening for tasks. I0405 11:05:41.048699 1 tasks_processing.go:69] worker 57 listening for tasks. I0405 11:05:41.048691 1 tasks_processing.go:69] worker 22 listening for tasks. I0405 11:05:41.048685 1 tasks_processing.go:69] worker 17 listening for tasks. I0405 11:05:41.048668 1 tasks_processing.go:69] worker 20 listening for tasks. I0405 11:05:41.048639 1 tasks_processing.go:69] worker 2 listening for tasks. I0405 11:05:41.048639 1 tasks_processing.go:69] worker 32 listening for tasks. I0405 11:05:41.048699 1 tasks_processing.go:69] worker 60 listening for tasks. I0405 11:05:41.048649 1 tasks_processing.go:69] worker 13 listening for tasks. I0405 11:05:41.048649 1 tasks_processing.go:69] worker 3 listening for tasks. I0405 11:05:41.048653 1 tasks_processing.go:69] worker 53 listening for tasks. I0405 11:05:41.048653 1 tasks_processing.go:69] worker 8 listening for tasks. I0405 11:05:41.048653 1 tasks_processing.go:69] worker 18 listening for tasks. I0405 11:05:41.048623 1 tasks_processing.go:69] worker 30 listening for tasks. I0405 11:05:41.048660 1 tasks_processing.go:69] worker 44 listening for tasks. I0405 11:05:41.048663 1 tasks_processing.go:69] worker 14 listening for tasks. I0405 11:05:41.048666 1 tasks_processing.go:69] worker 10 listening for tasks. I0405 11:05:41.048638 1 tasks_processing.go:69] worker 25 listening for tasks. I0405 11:05:41.048669 1 tasks_processing.go:69] worker 45 listening for tasks. I0405 11:05:41.048669 1 tasks_processing.go:69] worker 7 listening for tasks. I0405 11:05:41.048670 1 tasks_processing.go:69] worker 15 listening for tasks. I0405 11:05:41.048678 1 tasks_processing.go:69] worker 16 listening for tasks. I0405 11:05:41.048678 1 tasks_processing.go:69] worker 46 listening for tasks. I0405 11:05:41.048682 1 tasks_processing.go:69] worker 12 listening for tasks. I0405 11:05:41.048683 1 tasks_processing.go:69] worker 47 listening for tasks. I0405 11:05:41.048764 1 tasks_processing.go:71] worker 38 working on aggregated_monitoring_cr_names task. I0405 11:05:41.048768 1 tasks_processing.go:71] worker 39 working on version task. I0405 11:05:41.048770 1 tasks_processing.go:71] worker 21 working on openstack_dataplanedeployments task. I0405 11:05:41.048772 1 tasks_processing.go:71] worker 37 working on operators_pods_and_events task. I0405 11:05:41.048774 1 tasks_processing.go:71] worker 33 working on certificate_signing_requests task. I0405 11:05:41.048778 1 tasks_processing.go:71] worker 41 working on validating_webhook_configurations task. I0405 11:05:41.048779 1 tasks_processing.go:71] worker 47 working on active_alerts task. I0405 11:05:41.048790 1 tasks_processing.go:71] worker 59 working on dvo_metrics task. I0405 11:05:41.048790 1 tasks_processing.go:71] worker 52 working on oauths task. I0405 11:05:41.048803 1 tasks_processing.go:71] worker 36 working on networks task. W0405 11:05:41.048811 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:41.048816 1 tasks_processing.go:71] worker 8 working on operators task. I0405 11:05:41.048825 1 tasks_processing.go:71] worker 6 working on sap_pods task. I0405 11:05:41.048827 1 tasks_processing.go:71] worker 18 working on openstack_controlplanes task. I0405 11:05:41.048691 1 tasks_processing.go:69] worker 62 listening for tasks. I0405 11:05:41.048840 1 tasks_processing.go:71] worker 11 working on image_registries task. I0405 11:05:41.048840 1 tasks_processing.go:71] worker 50 working on sap_datahubs task. I0405 11:05:41.048845 1 tasks_processing.go:71] worker 4 working on machine_config_pools task. I0405 11:05:41.048849 1 tasks_processing.go:71] worker 58 working on openstack_dataplanenodesets task. I0405 11:05:41.048852 1 tasks_processing.go:71] worker 27 working on overlapping_namespace_uids task. I0405 11:05:41.048859 1 tasks_processing.go:71] worker 49 working on container_runtime_configs task. I0405 11:05:41.048770 1 tasks_processing.go:71] worker 19 working on ceph_cluster task. I0405 11:05:41.048965 1 tasks_processing.go:71] worker 31 working on machines task. I0405 11:05:41.049021 1 tasks_processing.go:71] worker 5 working on pod_network_connectivity_checks task. I0405 11:05:41.049043 1 tasks_processing.go:71] worker 29 working on proxies task. I0405 11:05:41.049057 1 tasks_processing.go:71] worker 7 working on metrics task. I0405 11:05:41.049072 1 tasks_processing.go:71] worker 42 working on openshift_machine_api_events task. I0405 11:05:41.049088 1 tasks_processing.go:71] worker 54 working on sap_config task. I0405 11:05:41.049093 1 tasks_processing.go:71] worker 14 working on infrastructures task. I0405 11:05:41.048840 1 tasks_processing.go:71] worker 24 working on machine_sets task. I0405 11:05:41.049031 1 tasks_processing.go:71] worker 9 working on silenced_alerts task. I0405 11:05:41.049278 1 tasks_processing.go:71] worker 45 working on node_logs task. I0405 11:05:41.049325 1 tasks_processing.go:71] worker 12 working on qemu_kubevirt_launcher_logs task. W0405 11:05:41.049077 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:41.048979 1 tasks_processing.go:71] worker 61 working on ingress_certificates task. I0405 11:05:41.049089 1 tasks_processing.go:71] worker 44 working on storage_cluster task. I0405 11:05:41.048773 1 tasks_processing.go:71] worker 40 working on feature_gates task. I0405 11:05:41.048823 1 tasks_processing.go:71] worker 47 working on machine_configs task. I0405 11:05:41.049281 1 tasks_processing.go:71] worker 32 working on schedulers task. I0405 11:05:41.048586 1 tasks_processing.go:69] worker 43 listening for tasks. I0405 11:05:41.048764 1 tasks_processing.go:71] worker 0 working on pdbs task. I0405 11:05:41.048764 1 tasks_processing.go:71] worker 23 working on nodes task. I0405 11:05:41.048764 1 tasks_processing.go:71] worker 34 working on support_secret task. I0405 11:05:41.048775 1 tasks_processing.go:71] worker 1 working on nodenetworkconfigurationpolicies task. I0405 11:05:41.048835 1 tasks_processing.go:71] worker 48 working on container_images task. I0405 11:05:41.048843 1 tasks_processing.go:71] worker 62 working on machine_autoscalers task. I0405 11:05:41.048988 1 tasks_processing.go:71] worker 26 working on service_accounts task. I0405 11:05:41.048992 1 tasks_processing.go:71] worker 35 working on install_plans task. I0405 11:05:41.050369 1 tasks_processing.go:71] worker 43 working on ingress task. I0405 11:05:41.048691 1 tasks_processing.go:69] worker 51 listening for tasks. I0405 11:05:41.050413 1 tasks_processing.go:71] worker 51 working on nodenetworkstates task. I0405 11:05:41.049085 1 tasks_processing.go:71] worker 30 working on clusterroles task. I0405 11:05:41.049252 1 tasks_processing.go:71] worker 28 working on cost_management_metrics_configs task. I0405 11:05:41.049271 1 tasks_processing.go:71] worker 10 working on cluster_apiserver task. I0405 11:05:41.049275 1 tasks_processing.go:71] worker 25 working on mutating_webhook_configurations task. I0405 11:05:41.049285 1 tasks_processing.go:71] worker 57 working on image_pruners task. I0405 11:05:41.049289 1 tasks_processing.go:71] worker 22 working on openstack_version task. I0405 11:05:41.049292 1 tasks_processing.go:71] worker 17 working on authentication task. I0405 11:05:41.049296 1 tasks_processing.go:71] worker 20 working on tsdb_status task. W0405 11:05:41.050931 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:41.049300 1 tasks_processing.go:71] worker 2 working on olm_operators task. W0405 11:05:41.049301 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:41.049305 1 tasks_processing.go:71] worker 46 working on image task. I0405 11:05:41.049309 1 tasks_processing.go:71] worker 15 working on monitoring_persistent_volumes task. I0405 11:05:41.049313 1 tasks_processing.go:71] worker 16 working on config_maps task. I0405 11:05:41.048693 1 tasks_processing.go:69] worker 56 listening for tasks. I0405 11:05:41.051278 1 tasks_processing.go:71] worker 56 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0405 11:05:41.049316 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 36.724µs to process 0 records I0405 11:05:41.049320 1 tasks_processing.go:71] worker 3 working on storage_classes task. I0405 11:05:41.051362 1 tasks_processing.go:71] worker 7 working on openshift_logging task. I0405 11:05:41.049323 1 tasks_processing.go:71] worker 60 working on jaegers task. I0405 11:05:41.049325 1 tasks_processing.go:71] worker 53 working on crds task. I0405 11:05:41.049326 1 tasks_processing.go:71] worker 13 working on lokistack task. I0405 11:05:41.048694 1 tasks_processing.go:69] worker 55 listening for tasks. I0405 11:05:41.051584 1 tasks_processing.go:74] worker 55 stopped. I0405 11:05:41.051294 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 405.552µs to process 0 records I0405 11:05:41.051602 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 19.153µs to process 0 records I0405 11:05:41.051613 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.759039ms to process 0 records I0405 11:05:41.051620 1 tasks_processing.go:74] worker 9 stopped. I0405 11:05:41.051625 1 tasks_processing.go:74] worker 20 stopped. I0405 11:05:41.053064 1 tasks_processing.go:74] worker 58 stopped. I0405 11:05:41.053079 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 4.206212ms to process 0 records I0405 11:05:41.053089 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 4.197663ms to process 0 records E0405 11:05:41.053095 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0405 11:05:41.053100 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 4.022583ms to process 0 records I0405 11:05:41.053122 1 tasks_processing.go:74] worker 49 stopped. I0405 11:05:41.053120 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 4.160824ms to process 0 records I0405 11:05:41.053128 1 tasks_processing.go:74] worker 19 stopped. I0405 11:05:41.053132 1 tasks_processing.go:74] worker 5 stopped. I0405 11:05:41.053228 1 tasks_processing.go:74] worker 63 stopped. E0405 11:05:41.053243 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0405 11:05:41.053260 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 4.620721ms to process 0 records I0405 11:05:41.053298 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0405 11:05:41.053312 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0405 11:05:41.053317 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0405 11:05:41.053320 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0405 11:05:41.053333 1 controller.go:489] The operator is still being initialized I0405 11:05:41.053341 1 controller.go:512] The operator is healthy I0405 11:05:41.060183 1 tasks_processing.go:74] worker 36 stopped. I0405 11:05:41.060372 1 recorder.go:75] Recording config/network with fingerprint=10cdc3b90693004923bfdce31cd096c75607c86f3bb4e3d14c49e5f05174a0a5 I0405 11:05:41.060385 1 gather.go:177] gatherer "clusterconfig" function "networks" took 11.365964ms to process 1 records I0405 11:05:41.060474 1 tasks_processing.go:74] worker 52 stopped. I0405 11:05:41.060565 1 recorder.go:75] Recording config/oauth with fingerprint=c87b03dd18bdcd4241ec73587b5fbfbc79c9201a92085928a1744dad1b119f37 I0405 11:05:41.060575 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 11.379003ms to process 1 records I0405 11:05:41.061167 1 tasks_processing.go:74] worker 4 stopped. I0405 11:05:41.061184 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 12.310116ms to process 0 records I0405 11:05:41.065202 1 tasks_processing.go:74] worker 21 stopped. I0405 11:05:41.065215 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 16.42452ms to process 0 records I0405 11:05:41.065243 1 tasks_processing.go:74] worker 6 stopped. I0405 11:05:41.065260 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 16.408744ms to process 0 records I0405 11:05:41.065278 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 16.421468ms to process 0 records E0405 11:05:41.065285 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0405 11:05:41.065291 1 gather.go:177] gatherer "clusterconfig" function "machines" took 16.252696ms to process 0 records I0405 11:05:41.065298 1 tasks_processing.go:74] worker 31 stopped. I0405 11:05:41.065302 1 tasks_processing.go:74] worker 18 stopped. I0405 11:05:41.068317 1 tasks_processing.go:74] worker 14 stopped. I0405 11:05:41.068808 1 recorder.go:75] Recording config/infrastructure with fingerprint=cafa9611d5122ee107793cf81d2e411f34b66db8aaebc17ab7b9c798e7926a83 I0405 11:05:41.068824 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 19.210044ms to process 1 records I0405 11:05:41.071190 1 tasks_processing.go:74] worker 62 stopped. I0405 11:05:41.071201 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 21.000493ms to process 0 records I0405 11:05:41.074607 1 tasks_processing.go:74] worker 42 stopped. I0405 11:05:41.074624 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 25.515063ms to process 0 records I0405 11:05:41.074682 1 tasks_processing.go:74] worker 0 stopped. I0405 11:05:41.074787 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=f0df1df5fc2a67f34b8755ec62d537bb3d0b19cbbd982956c15fcd655cffd8a4 I0405 11:05:41.074807 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=ada9ba042a62927fa381feb71982013f8a0643db88207a900b8363a6fc04a915 I0405 11:05:41.074823 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=81d9de73ead58d7c5e2ebb1bf085f4a2526b5d39f8b10f5a11125be13a87f356 I0405 11:05:41.074831 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 24.876803ms to process 3 records I0405 11:05:41.075170 1 tasks_processing.go:74] worker 45 stopped. I0405 11:05:41.075186 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 25.836959ms to process 0 records I0405 11:05:41.075334 1 tasks_processing.go:74] worker 32 stopped. I0405 11:05:41.075420 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=0fe74c796836b186a30ca5790db63289b12c81c0910c3f1befd51494c9fff98d I0405 11:05:41.075435 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 25.637996ms to process 1 records I0405 11:05:41.075653 1 tasks_processing.go:74] worker 11 stopped. I0405 11:05:41.076275 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=3ebeeb5bc32838e60fdb9bbd727f075ff505954b912531551e15508a4ce3cd5d I0405 11:05:41.076294 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 26.796833ms to process 1 records I0405 11:05:41.078209 1 tasks_processing.go:74] worker 7 stopped. I0405 11:05:41.078223 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 26.837361ms to process 0 records I0405 11:05:41.078341 1 tasks_processing.go:74] worker 41 stopped. I0405 11:05:41.078492 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=c41820f52c1be10a5a0e6b8aa7c3b647903ee71eab29922f97476c5b08c88410 I0405 11:05:41.078573 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=ed6e0e2b5a4267472e724bf86ae6b692802b8b8a07f046f264152a0a6e644678 I0405 11:05:41.078592 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=ee19ad007ee9ab065118fe5181474ca79ed4d5f633214eb2e9c75ddb2e1fe845 I0405 11:05:41.078628 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=91a2c69f6281f91b81ec3a65ab65210589d991c241ae75946b924399759dac5f I0405 11:05:41.078655 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=70b897d3585fce0b4c426fbf367862ce64b59f7d75c8ae6431b57e466b847ad9 I0405 11:05:41.078684 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=d0c9ea49c3fb09038f547e2c170bd74fb99f9d2f7070e60824268eadcf33f885 I0405 11:05:41.078710 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=518fefbd427097cb00e0b005ac2aac8b3455cf2f457c43916fe9810e78ec8828 I0405 11:05:41.078744 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=262c9477a431eba99de2ba25cd85f4a554958a635f107f0032c050e68f502cfb I0405 11:05:41.078769 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=2a2cbd915db1b0ac019d925bb81c2cb7f5cb8eab84af254ca6d55cb33526e8b8 I0405 11:05:41.078791 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=735c8fedca264e2e18ee04090eea5261e4cefb7154535ab8e28f9e519b385061 I0405 11:05:41.078818 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=e22c998caeda567ea6757089b95d9a6734b7e353c69b9010b4447d4777fbfe68 I0405 11:05:41.078826 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 29.549515ms to process 11 records W0405 11:05:41.078853 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 11:05:41.078924 1 recorder.go:75] Recording config/ingress with fingerprint=29eff77453e327c85c01fa55bfb3d325a8fd6bd8d7f151521d2b4db98dec2f8b I0405 11:05:41.078927 1 tasks_processing.go:74] worker 43 stopped. I0405 11:05:41.078931 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 28.286747ms to process 1 records I0405 11:05:41.078940 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0405 11:05:41.078946 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 29.833251ms to process 1 records I0405 11:05:41.078951 1 tasks_processing.go:74] worker 27 stopped. I0405 11:05:41.081579 1 tasks_processing.go:74] worker 46 stopped. I0405 11:05:41.081673 1 recorder.go:75] Recording config/image with fingerprint=88bed86c272518233e89cd155b23b73774d5f15fb93490ffc04959da23bebc80 I0405 11:05:41.081691 1 gather.go:177] gatherer "clusterconfig" function "image" took 30.532123ms to process 1 records I0405 11:05:41.081778 1 tasks_processing.go:74] worker 17 stopped. I0405 11:05:41.081833 1 recorder.go:75] Recording config/authentication with fingerprint=1be1bc4c66afc6748cfc1979d865eb7a43a84fa80988ee4d46d17fc943e8c1d6 I0405 11:05:41.081844 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 30.778071ms to process 1 records I0405 11:05:41.081922 1 recorder.go:75] Recording config/featuregate with fingerprint=9149e64f69bb6c6be7015486f4cc60a2c7a0983a3abeeca0a9ad60548ce735dd I0405 11:05:41.081931 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 32.083217ms to process 1 records I0405 11:05:41.081937 1 tasks_processing.go:74] worker 40 stopped. I0405 11:05:41.081943 1 tasks_processing.go:74] worker 33 stopped. I0405 11:05:41.081952 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 33.160156ms to process 0 records I0405 11:05:41.082976 1 tasks_processing.go:74] worker 10 stopped. I0405 11:05:41.083461 1 recorder.go:75] Recording config/apiserver with fingerprint=40c4084601e59d244490df68dfac8c6de50a66e54ff9269ba81fbc1e5f19beb3 I0405 11:05:41.083483 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 32.360921ms to process 1 records I0405 11:05:41.083517 1 tasks_processing.go:74] worker 1 stopped. I0405 11:05:41.083931 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 33.44817ms to process 0 records I0405 11:05:41.083953 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 34.007153ms to process 0 records I0405 11:05:41.083959 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 33.156369ms to process 0 records I0405 11:05:41.083964 1 tasks_processing.go:74] worker 22 stopped. I0405 11:05:41.083964 1 tasks_processing.go:74] worker 44 stopped. I0405 11:05:41.083991 1 tasks_processing.go:74] worker 51 stopped. I0405 11:05:41.084011 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 33.568263ms to process 0 records I0405 11:05:41.084018 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 33.416178ms to process 0 records I0405 11:05:41.084026 1 tasks_processing.go:74] worker 28 stopped. I0405 11:05:41.084077 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0405 11:05:41.084090 1 tasks_processing.go:74] worker 3 stopped. I0405 11:05:41.084102 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0405 11:05:41.084203 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=c9c53a7e225430b4f73b6ff4e019c132d4b1a11e6a083654847be8f490bf6944 I0405 11:05:41.084220 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=9394c529b095cd7b8a0a28ea25bbd9b201b8b164304381c6c6a0b542a4dec3a2 I0405 11:05:41.084229 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 32.781635ms to process 2 records W0405 11:05:41.084230 1 operator.go:288] started I0405 11:05:41.084250 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0405 11:05:41.084365 1 tasks_processing.go:74] worker 13 stopped. I0405 11:05:41.084375 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 32.856034ms to process 0 records I0405 11:05:41.084795 1 tasks_processing.go:74] worker 50 stopped. I0405 11:05:41.084808 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 35.943804ms to process 0 records I0405 11:05:41.084824 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 33.360286ms to process 0 records I0405 11:05:41.084829 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 35.65401ms to process 0 records I0405 11:05:41.084833 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 35.717128ms to process 0 records I0405 11:05:41.084836 1 tasks_processing.go:74] worker 60 stopped. I0405 11:05:41.084840 1 tasks_processing.go:74] worker 54 stopped. I0405 11:05:41.084845 1 tasks_processing.go:74] worker 24 stopped. I0405 11:05:41.085688 1 tasks_processing.go:74] worker 34 stopped. E0405 11:05:41.085705 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0405 11:05:41.085719 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 35.747213ms to process 0 records I0405 11:05:41.092164 1 tasks_processing.go:74] worker 29 stopped. I0405 11:05:41.092285 1 recorder.go:75] Recording config/proxy with fingerprint=2dc2df5a71baa97ba1a6cb0ca11ce670b43fdaafa23fcf18ce752f71d58f9b9c I0405 11:05:41.092303 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 43.10601ms to process 1 records I0405 11:05:41.097344 1 gather_logs.go:145] no pods in namespace were found I0405 11:05:41.097358 1 tasks_processing.go:74] worker 12 stopped. I0405 11:05:41.097368 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 47.992553ms to process 0 records I0405 11:05:41.097629 1 tasks_processing.go:74] worker 38 stopped. I0405 11:05:41.097638 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 48.855724ms to process 0 records I0405 11:05:41.097714 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=86139ccd2a67c79e69e2d9444699aa8f71c61834cea988e8a753ce3f4a8c03f3 I0405 11:05:41.097732 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 46.911957ms to process 1 records I0405 11:05:41.097741 1 tasks_processing.go:74] worker 57 stopped. I0405 11:05:41.097840 1 tasks_processing.go:74] worker 25 stopped. I0405 11:05:41.097942 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=08d14eb3de613d1b65a176a5eff4575d8d7ad94b91fbc72f1056e8199e5841b6 I0405 11:05:41.097987 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=64d03d1f95e0ee740895ca6e2296a24858292c0b2aa9604f7db9aecffa9a8394 I0405 11:05:41.098011 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=0936f852822b2182f63472ae662892fd36b48222e0d974e3aa831837c555c74b I0405 11:05:41.098019 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 47.141756ms to process 3 records I0405 11:05:41.098025 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 46.848067ms to process 0 records I0405 11:05:41.098033 1 tasks_processing.go:74] worker 15 stopped. I0405 11:05:41.098176 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=9d989b78d5ef8d930167e6493b270e1d7fd8eadc4668c647d474bd1be08ec452 I0405 11:05:41.098186 1 tasks_processing.go:74] worker 30 stopped. I0405 11:05:41.098257 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=8f79a145c822a0699661604c366ff4d7f193c33df58c583cf7098db664002cee I0405 11:05:41.098263 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 47.599955ms to process 2 records I0405 11:05:41.098613 1 tasks_processing.go:74] worker 53 stopped. I0405 11:05:41.098925 1 controller.go:212] Source scaController *sca.Controller is not ready I0405 11:05:41.098937 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0405 11:05:41.098943 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0405 11:05:41.098946 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0405 11:05:41.098951 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0405 11:05:41.098971 1 controller.go:489] The operator is still being initialized I0405 11:05:41.098984 1 controller.go:512] The operator is healthy I0405 11:05:41.099417 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=fba34991a005dc213e129a87a302dc7ba54c69d4a3c3f33df521addace38808f I0405 11:05:41.100326 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7d833e19521ee3abecf71725f2d5be72436c63f6b99b28722578d309cf807cfe I0405 11:05:41.100392 1 gather.go:177] gatherer "clusterconfig" function "crds" took 47.121139ms to process 2 records I0405 11:05:41.100521 1 recorder.go:75] Recording config/olm_operators with fingerprint=06bee57102987a6ed44ce201e41369b728e253dfffe5047fa0c7431d19089ee3 I0405 11:05:41.100542 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 48.082688ms to process 1 records I0405 11:05:41.100561 1 tasks_processing.go:74] worker 2 stopped. I0405 11:05:41.101881 1 prometheus_rules.go:88] Prometheus rules successfully created I0405 11:05:41.103562 1 tasks_processing.go:74] worker 48 stopped. I0405 11:05:41.104947 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-m6v6t with fingerprint=18c50df5efc8592809fd00e6a52c32737bd6b72472281f91eb8b2cc209ef5ae5 I0405 11:05:41.105053 1 recorder.go:75] Recording config/running_containers with fingerprint=16bae3840a060863b1bd4b6d75e4234492869f0a4810bdd279e2943caa9b2e74 I0405 11:05:41.105073 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 53.518978ms to process 2 records E0405 11:05:41.108902 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27f2d26656-3c79-44fb-86cf-144821cdc275%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.11:50774->172.30.0.10:53: read: connection refused I0405 11:05:41.108928 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27f2d26656-3c79-44fb-86cf-144821cdc275%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.11:50774->172.30.0.10:53: read: connection refused I0405 11:05:41.110664 1 tasks_processing.go:74] worker 56 stopped. I0405 11:05:41.110676 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 59.375789ms to process 0 records I0405 11:05:41.113599 1 tasks_processing.go:74] worker 16 stopped. E0405 11:05:41.113611 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0405 11:05:41.113616 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0405 11:05:41.113620 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0405 11:05:41.113628 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0405 11:05:41.113650 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0405 11:05:41.113657 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0405 11:05:41.113661 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0405 11:05:41.113665 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0405 11:05:41.113702 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0405 11:05:41.113709 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0405 11:05:41.113714 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 62.38624ms to process 7 records I0405 11:05:41.117013 1 tasks_processing.go:74] worker 61 stopped. E0405 11:05:41.117026 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0405 11:05:41.117032 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pfutctdbuh0ml72j189oi2h49ufv2jo-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pfutctdbuh0ml72j189oi2h49ufv2jo-primary-cert-bundle-secret" not found I0405 11:05:41.117074 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=df9ec817aad106bb73616ef8b5edd2774c2dfca99edca0c2578a06af0044dc93 I0405 11:05:41.117084 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 67.534408ms to process 1 records I0405 11:05:41.117101 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0405 11:05:41.117127 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0405 11:05:41.117198 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0405 11:05:41.125095 1 base_controller.go:82] Caches are synced for ConfigController I0405 11:05:41.125116 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0405 11:05:41.128956 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 11:05:41.171194 1 tasks_processing.go:74] worker 23 stopped. I0405 11:05:41.171602 1 recorder.go:75] Recording config/node/ip-10-0-1-198.ec2.internal with fingerprint=fa8e86a1e1347d3b71364e76dd571b9fca6ece8198e8543cffd909167688c966 I0405 11:05:41.171621 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 121.268258ms to process 1 records I0405 11:05:41.172814 1 tasks_processing.go:74] worker 47 stopped. I0405 11:05:41.172835 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0405 11:05:41.172844 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 123.200859ms to process 1 records I0405 11:05:41.178613 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0405 11:05:41.181748 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.11:55849->172.30.0.10:53: read: connection refused I0405 11:05:41.181760 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.11:55849->172.30.0.10:53: read: connection refused I0405 11:05:41.184782 1 base_controller.go:82] Caches are synced for LoggingSyncer I0405 11:05:41.184792 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0405 11:05:41.184947 1 tasks_processing.go:74] worker 39 stopped. I0405 11:05:41.185326 1 recorder.go:75] Recording config/version with fingerprint=5e2c17779e8e7f9b194a2ead243cec1d42d8e391f7fd7f551dc06145de3ac892 I0405 11:05:41.185343 1 recorder.go:75] Recording config/id with fingerprint=0cec12c3024123a7c63b37e407424caffc096ca1a168071d4bedb8238887e284 I0405 11:05:41.185353 1 gather.go:177] gatherer "clusterconfig" function "version" took 136.16313ms to process 2 records W0405 11:05:42.079013 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 11:05:42.282174 1 gather_cluster_operator_pods_and_events.go:121] Found 12 pods with 14 containers I0405 11:05:42.282187 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1797558 bytes I0405 11:05:42.282998 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-k7zvk pod in namespace openshift-dns (previous: false). I0405 11:05:42.509471 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-k7zvk pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-k7zvk\" is waiting to start: ContainerCreating" I0405 11:05:42.509490 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-k7zvk\" is waiting to start: ContainerCreating" I0405 11:05:42.509498 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-k7zvk pod in namespace openshift-dns (previous: false). I0405 11:05:42.514219 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0405 11:05:42.689190 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-k7zvk pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-k7zvk\" is waiting to start: ContainerCreating" I0405 11:05:42.689208 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-k7zvk\" is waiting to start: ContainerCreating" I0405 11:05:42.689218 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-x7rmd pod in namespace openshift-dns (previous: false). I0405 11:05:42.887624 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 11:05:42.887672 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6fd7f6c764-rmvff pod in namespace openshift-image-registry (previous: false). W0405 11:05:43.078900 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 11:05:43.081145 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 11:05:43.081193 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-847bdbd4bc-7kcjz pod in namespace openshift-image-registry (previous: false). I0405 11:05:43.115209 1 tasks_processing.go:74] worker 8 stopped. I0405 11:05:43.115257 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=59b888c3feb14f8c5a9fbfe6f1b28d7e1634f862c2e62deb60dd0109c563f881 I0405 11:05:43.115291 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=c19f131c78299491412120fd30dc807b866f227847d90502e329d6e2ec562a4d I0405 11:05:43.115320 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0405 11:05:43.115349 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=f716a610180dfa330c244fec09b2457738193ba71ff46dd7b6da20c99e367669 I0405 11:05:43.115365 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0405 11:05:43.115387 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=76e79ec9588e5669e3721adccef2921c43c0e3329a665b7a751627302bbb5909 I0405 11:05:43.115432 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=fa54b4232796a9046aeea17581f70b794154645a4025cb90196f895d136b85a9 I0405 11:05:43.115456 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=5ea3883b443d21c99bb28ffd8eda2079cb18c8d8de5a0b65aa97afce15159a96 I0405 11:05:43.115469 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=09c5777c727bd6563bf1b58d56b7197229bcd6fc12e35b07406d479a14dadc45 I0405 11:05:43.115485 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=b9de666718c62e4af58bfc2f1aae4c23f4ff0611c776f1ffd55391257b2534b0 I0405 11:05:43.115494 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0405 11:05:43.115510 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=55e4753fbc44de8fb398fac9d7d8009daae068f5364ab31f7e1ecc4fae4cc549 I0405 11:05:43.115522 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0405 11:05:43.115537 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=7b977a1bbdf4f968278c10530ca8f57ffd170030a95ef56e284bd0ad0b8d1066 I0405 11:05:43.115546 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0405 11:05:43.115565 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=633bd8a77fbbe75f59f4137b4ac5f790df8b3bf439bf90a72ea31e96b2af8d5a I0405 11:05:43.115573 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0405 11:05:43.115590 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=48dc2deb9a1af5b9b5555d89876e1fc630507ad33b46a70200ac2987223ea70d I0405 11:05:43.115712 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=9a169454129e932be5c8ab02910ef50dd2c387e4900b747a8aab8a10df3b57f4 I0405 11:05:43.115722 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0405 11:05:43.115728 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0405 11:05:43.115749 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0405 11:05:43.115772 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=ffe129cc1bd6a00a345fe4efb093ed5eedc57477582b567ff20505ad03b2fd16 I0405 11:05:43.115797 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=0bf5ed475925b431698e1e626522d20e05f8b6df92ca0fd0ade57ebe0228ff92 I0405 11:05:43.115807 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0405 11:05:43.115823 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=d64bcb4220aaed2144df09ac4a5f3ed9674cd9bc32591a308a9730d3bb6acd00 I0405 11:05:43.115833 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0405 11:05:43.115846 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=01e47281d041237ddefb1b4cc9671e3c93c2d4d01807e548c8ae0753ec30fc28 I0405 11:05:43.115860 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=d42cc6c20be7a3bd4b223499ebbc32c929f57c0d155eceb46ba1a0ef6dd59cf8 I0405 11:05:43.115875 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=e75bda2fe52549e30d9c9e99cf082b3d154078f6cc0b31883d90b2dd5039e7b8 I0405 11:05:43.115891 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=31496a2391ffdf7273720e5afb96d71ba4a5fb7b49f656fcada1fb7712ed74ad I0405 11:05:43.115911 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=463446a241d3817c896b85238766a7e823b299bb2ee4441aa139630f05cfcf49 I0405 11:05:43.115919 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0405 11:05:43.115942 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=bbb7710bda8704c516cd617e00e7725071c23d71fca78b239e58c043d94c3447 I0405 11:05:43.115958 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0405 11:05:43.115966 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0405 11:05:43.115973 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.066372163s to process 36 records I0405 11:05:43.287242 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-847bdbd4bc-7kcjz pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-847bdbd4bc-7kcjz\" is waiting to start: ContainerCreating" I0405 11:05:43.287256 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-847bdbd4bc-7kcjz\" is waiting to start: ContainerCreating" I0405 11:05:43.287283 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-847bdbd4bc-ds2sw pod in namespace openshift-image-registry (previous: false). I0405 11:05:43.481530 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 11:05:43.481544 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-w8mw5 pod in namespace openshift-image-registry (previous: false). I0405 11:05:43.687734 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 11:05:43.687748 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-657668565c-mp4vp pod in namespace openshift-ingress (previous: false). I0405 11:05:43.915173 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 11:05:43.915202 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-657668565c-q4mhk pod in namespace openshift-ingress (previous: false). W0405 11:05:44.078823 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 11:05:44.089157 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-657668565c-q4mhk pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-657668565c-q4mhk\" is waiting to start: ContainerCreating" I0405 11:05:44.089171 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-657668565c-q4mhk\" is waiting to start: ContainerCreating" I0405 11:05:44.089181 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6588d4cb8f-q7mzh pod in namespace openshift-ingress (previous: false). I0405 11:05:44.300753 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6588d4cb8f-q7mzh pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6588d4cb8f-q7mzh\" is waiting to start: ContainerCreating" I0405 11:05:44.300771 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6588d4cb8f-q7mzh\" is waiting to start: ContainerCreating" I0405 11:05:44.300828 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-6fgq8 pod in namespace openshift-ingress-canary (previous: false). I0405 11:05:44.490626 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-6fgq8 pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-6fgq8\" is waiting to start: ContainerCreating" I0405 11:05:44.490642 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-6fgq8\" is waiting to start: ContainerCreating" I0405 11:05:44.490653 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-nzwmg pod in namespace openshift-kube-storage-version-migrator (previous: false). I0405 11:05:44.699240 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-nzwmg pod in namespace openshift-kube-storage-version-migrator (previous: false). I0405 11:05:44.896309 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-5vs67 pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). W0405 11:05:45.079266 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 11:05:45.088252 1 tasks_processing.go:74] worker 37 stopped. I0405 11:05:45.088324 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=2f977c90352fb42e045ebbfde4e17938b7925c16c7e8109208dffc094f4f633a I0405 11:05:45.088349 1 recorder.go:75] Recording events/openshift-dns with fingerprint=62cb99694f2b26cd38840380e3d9c17dd15ebf177d2584f892b8adf1ce401dab I0405 11:05:45.088410 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=f0dac00a893b4494212ad421c8273b36694ca926d1094204d5c5dae535ad84a0 I0405 11:05:45.088437 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=3c409c8e6455262db5466ec851a280a56b1bf3bc277662eaf3f3cc355ced79cc I0405 11:05:45.088478 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=c0c7bd2be1f1486954d7875d759ed6784440fbe0e0badd4778a531284aab094f I0405 11:05:45.088490 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=8b881a71ff169dd4a0f84a7c1072fc4af383fb18b84d6908b3b59ef73af31e8e I0405 11:05:45.088502 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=d75b5fc5d531dd2f34a23da9b7a80ee625534fd18d176d8bd963b84bf9cb4c6e I0405 11:05:45.088556 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=a47bb6ac399093f430725e4e716d0f8906c4bde18d62294ceae2d2bb6a154f3e I0405 11:05:45.088714 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-k7zvk with fingerprint=714d6e701cd29bbd19465c1effb4d0b325875aa47c4872049706d8998e1a35ae I0405 11:05:45.088813 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-6fd7f6c764-rmvff with fingerprint=a221efca8295b01e243ecb268d61aac5ad0510ef79f66a78d9f06d39ff72f1e7 I0405 11:05:45.088952 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-847bdbd4bc-7kcjz with fingerprint=3636f9801feb213673e7ad077d0fe69c300988de27a3f0293eeb380f868b74bd I0405 11:05:45.089077 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-847bdbd4bc-ds2sw with fingerprint=3477dc05f848c44dba3bfea4cabd3c171e04f47df759656480ea4ac46d2bf855 I0405 11:05:45.089163 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-6fgq8 with fingerprint=f340759d6a7c3541d51390866aec321dbbcfbc32e922dba2aad4185d1e300285 I0405 11:05:45.089180 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-nzwmg/migrator_current.log with fingerprint=4a07c014ec0480fc3a38c372eb251d149af29e878af7c5d655b67ce7ee724cc9 I0405 11:05:45.089186 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-nzwmg/graceful-termination_current.log with fingerprint=65f44fe938dec0aa5dddcff96dce200c0f2333c79ba1423376d21435b3fc7e9c I0405 11:05:45.089252 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-5vs67/kube-storage-version-migrator-operator_current.log with fingerprint=caf0736950122f1f646fe6e97afe7f0bf76ec198298843f251bcf412fb838060 I0405 11:05:45.089262 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.039465026s to process 16 records W0405 11:05:46.078134 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0405 11:05:46.078163 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0405 11:05:46.078175 1 tasks_processing.go:74] worker 59 stopped. E0405 11:05:46.078185 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0405 11:05:46.078196 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0405 11:05:46.078210 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0405 11:05:46.078224 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.029375021s to process 1 records I0405 11:05:53.009278 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 11:05:53.043192 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 11:05:53.209569 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 11:05:53.889054 1 tasks_processing.go:74] worker 35 stopped. I0405 11:05:53.889122 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0405 11:05:53.889138 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.838664683s to process 1 records I0405 11:05:54.656248 1 tasks_processing.go:74] worker 26 stopped. I0405 11:05:54.656515 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=84ad0e43752ccb8a509dca2afcd9549ee386cc6542bb678120e8c4271861dfb6 I0405 11:05:54.656531 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605977341s to process 1 records E0405 11:05:54.656585 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.608s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0405 11:05:54.657688 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0405 11:05:54.657703 1 periodic.go:209] Running workloads gatherer I0405 11:05:54.657717 1 tasks_processing.go:45] number of workers: 2 I0405 11:05:54.657723 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 11:05:54.657727 1 tasks_processing.go:71] worker 1 working on workload_info task. I0405 11:05:54.657791 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 11:05:54.657860 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0405 11:05:54.675731 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0405 11:05:54.683461 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0405 11:05:54.683814 1 tasks_processing.go:74] worker 0 stopped. I0405 11:05:54.683833 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.937912ms to process 0 records I0405 11:05:54.690422 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (7ms) I0405 11:05:54.697610 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (7ms) I0405 11:05:54.704818 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (7ms) I0405 11:05:54.711883 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (7ms) I0405 11:05:54.718710 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (7ms) I0405 11:05:54.726215 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (7ms) I0405 11:05:54.732872 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (7ms) I0405 11:05:54.739577 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (7ms) I0405 11:05:54.747052 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (7ms) I0405 11:05:54.783962 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (37ms) I0405 11:05:54.884081 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0405 11:05:54.984691 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0405 11:05:55.084248 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0405 11:05:55.183388 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (99ms) I0405 11:05:55.283872 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0405 11:05:55.384690 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0405 11:05:55.483662 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (99ms) I0405 11:05:55.584422 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (101ms) I0405 11:05:55.683693 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (99ms) I0405 11:05:55.784900 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (101ms) I0405 11:05:55.884135 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (99ms) I0405 11:05:55.983606 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (99ms) I0405 11:05:56.085877 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (102ms) I0405 11:05:56.183633 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (98ms) I0405 11:05:56.283644 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0405 11:05:56.383925 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0405 11:05:56.484204 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0405 11:05:56.584612 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0405 11:05:56.683897 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (99ms) I0405 11:05:56.784056 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0405 11:05:56.784089 1 tasks_processing.go:74] worker 1 stopped. E0405 11:05:56.784103 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0405 11:05:56.784429 1 recorder.go:75] Recording config/workload_info with fingerprint=08d59fbb44642567aeaf27122d597ac17cb81e2007b2e3512ec3f670226cff96 I0405 11:05:56.784450 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.126354597s to process 1 records E0405 11:05:56.784486 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.126s with: function \"workload_info\" failed with an error" I0405 11:05:56.785587 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0405 11:05:56.785599 1 periodic.go:209] Running conditional gatherer I0405 11:05:56.790462 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0405 11:05:56.796833 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.11:45319->172.30.0.10:53: read: connection refused E0405 11:05:56.797090 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 11:05:56.797176 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0405 11:05:56.803262 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0405 11:05:56.803275 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803280 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803283 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803286 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803289 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803292 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803296 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803299 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 11:05:56.803302 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0405 11:05:56.803316 1 tasks_processing.go:45] number of workers: 3 I0405 11:05:56.803327 1 tasks_processing.go:69] worker 2 listening for tasks. I0405 11:05:56.803330 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0405 11:05:56.803342 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 11:05:56.803351 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 11:05:56.803361 1 tasks_processing.go:74] worker 1 stopped. I0405 11:05:56.803358 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0405 11:05:56.803355 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0405 11:05:56.803387 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0405 11:05:56.803400 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 746ns to process 1 records I0405 11:05:56.803459 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0405 11:05:56.803470 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.083µs to process 1 records I0405 11:05:56.803478 1 tasks_processing.go:74] worker 2 stopped. I0405 11:05:56.803630 1 tasks_processing.go:74] worker 0 stopped. I0405 11:05:56.803643 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 259.274µs to process 0 records I0405 11:05:56.803674 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.11:45319->172.30.0.10:53: read: connection refused I0405 11:05:56.803693 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0405 11:05:56.824935 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=04289d307ef371852db280b0b5f4f4a82496c43a87af0d1e0ad9a16298c6960c I0405 11:05:56.825052 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-04-05-110556.tar.gz I0405 11:05:56.832024 1 diskrecorder.go:51] Wrote 110 records to disk in 7ms I0405 11:05:56.832052 1 periodic.go:278] Gathering cluster info every 2h0m0s I0405 11:05:56.832065 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0405 11:06:04.972650 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 11:06:49.810642 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="f937bf1bb7274d1971a00b040f187998c9fa13670af449f08a6046dc3cda49e4") W0405 11:06:49.810672 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0405 11:06:49.810717 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="ff1ba7d8c3285e6dfb1b26131d13a508a1529fab9f2132e29c0ff2ab64eb6873") I0405 11:06:49.810735 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0405 11:06:49.810755 1 base_controller.go:181] Shutting down ConfigController ... I0405 11:06:49.810772 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0405 11:06:49.810776 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0405 11:06:49.810790 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="0146c96f52531bbd4956c7b801b7505ae659a6b40eaae15d7ed49ee532f2f354") I0405 11:06:49.810800 1 periodic.go:170] Shutting down I0405 11:06:49.810815 1 base_controller.go:181] Shutting down LoggingSyncer ... I0405 11:06:49.810827 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0405 11:06:49.810833 1 base_controller.go:113] All ConfigController workers have been terminated