W0506 03:04:05.859864 1 cmd.go:257] Using insecure, self-signed certificates I0506 03:04:06.578885 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:06.579238 1 observer_polling.go:159] Starting file observer I0506 03:04:06.957223 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0506 03:04:06.957436 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0506 03:04:06.957961 1 secure_serving.go:57] Forcing use of http/1.1 only W0506 03:04:06.957980 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0506 03:04:06.957985 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0506 03:04:06.957989 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0506 03:04:06.957991 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0506 03:04:06.957993 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0506 03:04:06.957996 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0506 03:04:06.958000 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0506 03:04:06.964818 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0506 03:04:06.964876 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"036a5d91-afd7-46b1-a315-004535d274e4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0506 03:04:06.969070 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0506 03:04:06.969094 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0506 03:04:06.969089 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0506 03:04:06.969096 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0506 03:04:06.969125 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0506 03:04:06.969129 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0506 03:04:06.969458 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1008527116/tls.crt::/tmp/serving-cert-1008527116/tls.key" I0506 03:04:06.969742 1 secure_serving.go:213] Serving securely on [::]:8443 I0506 03:04:06.969770 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0506 03:04:06.973510 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0506 03:04:06.973554 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0506 03:04:06.973689 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0506 03:04:06.978770 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0506 03:04:06.978785 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0506 03:04:06.985229 1 secretconfigobserver.go:119] support secret does not exist I0506 03:04:06.990065 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0506 03:04:06.994881 1 secretconfigobserver.go:119] support secret does not exist I0506 03:04:06.997099 1 recorder.go:161] Pruning old reports every 4h48m50s, max age is 288h0m0s I0506 03:04:07.003749 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0506 03:04:07.003768 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0506 03:04:07.003776 1 periodic.go:209] Running clusterconfig gatherer I0506 03:04:07.003782 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0506 03:04:07.003798 1 insightsreport.go:296] Starting report retriever I0506 03:04:07.003806 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0506 03:04:07.003814 1 tasks_processing.go:45] number of workers: 64 I0506 03:04:07.003833 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 03:04:07.003845 1 tasks_processing.go:69] worker 14 listening for tasks. I0506 03:04:07.003846 1 tasks_processing.go:69] worker 7 listening for tasks. I0506 03:04:07.003853 1 tasks_processing.go:71] worker 14 working on image task. I0506 03:04:07.003854 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 03:04:07.003857 1 tasks_processing.go:71] worker 7 working on cost_management_metrics_configs task. I0506 03:04:07.003861 1 tasks_processing.go:69] worker 15 listening for tasks. I0506 03:04:07.003866 1 tasks_processing.go:69] worker 5 listening for tasks. I0506 03:04:07.003866 1 tasks_processing.go:69] worker 4 listening for tasks. I0506 03:04:07.003876 1 tasks_processing.go:69] worker 6 listening for tasks. I0506 03:04:07.003876 1 tasks_processing.go:69] worker 16 listening for tasks. I0506 03:04:07.003882 1 tasks_processing.go:69] worker 17 listening for tasks. I0506 03:04:07.003881 1 tasks_processing.go:69] worker 2 listening for tasks. I0506 03:04:07.003882 1 tasks_processing.go:69] worker 45 listening for tasks. I0506 03:04:07.003887 1 tasks_processing.go:69] worker 3 listening for tasks. I0506 03:04:07.003891 1 tasks_processing.go:69] worker 18 listening for tasks. I0506 03:04:07.003895 1 tasks_processing.go:69] worker 60 listening for tasks. I0506 03:04:07.003898 1 tasks_processing.go:69] worker 19 listening for tasks. I0506 03:04:07.003898 1 tasks_processing.go:69] worker 27 listening for tasks. I0506 03:04:07.003899 1 tasks_processing.go:69] worker 55 listening for tasks. I0506 03:04:07.003903 1 tasks_processing.go:69] worker 56 listening for tasks. I0506 03:04:07.003910 1 tasks_processing.go:69] worker 51 listening for tasks. I0506 03:04:07.003906 1 tasks_processing.go:69] worker 28 listening for tasks. I0506 03:04:07.003916 1 tasks_processing.go:69] worker 21 listening for tasks. I0506 03:04:07.003909 1 tasks_processing.go:69] worker 20 listening for tasks. I0506 03:04:07.003913 1 tasks_processing.go:69] worker 29 listening for tasks. I0506 03:04:07.003925 1 tasks_processing.go:69] worker 22 listening for tasks. I0506 03:04:07.003914 1 tasks_processing.go:69] worker 57 listening for tasks. I0506 03:04:07.003926 1 tasks_processing.go:69] worker 59 listening for tasks. I0506 03:04:07.003930 1 tasks_processing.go:69] worker 58 listening for tasks. I0506 03:04:07.003936 1 tasks_processing.go:69] worker 54 listening for tasks. I0506 03:04:07.003928 1 tasks_processing.go:69] worker 53 listening for tasks. I0506 03:04:07.003940 1 tasks_processing.go:69] worker 48 listening for tasks. I0506 03:04:07.003914 1 tasks_processing.go:69] worker 50 listening for tasks. I0506 03:04:07.003947 1 tasks_processing.go:69] worker 49 listening for tasks. I0506 03:04:07.003950 1 tasks_processing.go:69] worker 8 listening for tasks. I0506 03:04:07.003942 1 tasks_processing.go:69] worker 25 listening for tasks. I0506 03:04:07.003954 1 tasks_processing.go:69] worker 39 listening for tasks. I0506 03:04:07.003951 1 tasks_processing.go:69] worker 24 listening for tasks. I0506 03:04:07.003956 1 tasks_processing.go:69] worker 42 listening for tasks. I0506 03:04:07.003957 1 tasks_processing.go:69] worker 41 listening for tasks. I0506 03:04:07.003962 1 tasks_processing.go:69] worker 43 listening for tasks. I0506 03:04:07.003964 1 tasks_processing.go:69] worker 38 listening for tasks. I0506 03:04:07.003969 1 tasks_processing.go:71] worker 0 working on cluster_apiserver task. I0506 03:04:07.003971 1 tasks_processing.go:69] worker 12 listening for tasks. I0506 03:04:07.003970 1 tasks_processing.go:69] worker 44 listening for tasks. I0506 03:04:07.003973 1 tasks_processing.go:69] worker 11 listening for tasks. I0506 03:04:07.003924 1 tasks_processing.go:69] worker 30 listening for tasks. I0506 03:04:07.003927 1 tasks_processing.go:69] worker 46 listening for tasks. I0506 03:04:07.003933 1 tasks_processing.go:69] worker 23 listening for tasks. I0506 03:04:07.003932 1 tasks_processing.go:69] worker 31 listening for tasks. I0506 03:04:07.003933 1 tasks_processing.go:69] worker 47 listening for tasks. I0506 03:04:07.003941 1 tasks_processing.go:69] worker 32 listening for tasks. I0506 03:04:07.003990 1 tasks_processing.go:69] worker 13 listening for tasks. I0506 03:04:07.003944 1 tasks_processing.go:69] worker 10 listening for tasks. I0506 03:04:07.003947 1 tasks_processing.go:69] worker 33 listening for tasks. I0506 03:04:07.003948 1 tasks_processing.go:69] worker 62 listening for tasks. I0506 03:04:07.003920 1 tasks_processing.go:69] worker 52 listening for tasks. I0506 03:04:07.003956 1 tasks_processing.go:69] worker 40 listening for tasks. I0506 03:04:07.003957 1 tasks_processing.go:69] worker 36 listening for tasks. I0506 03:04:07.003960 1 tasks_processing.go:69] worker 37 listening for tasks. I0506 03:04:07.003952 1 tasks_processing.go:69] worker 9 listening for tasks. I0506 03:04:07.003962 1 tasks_processing.go:69] worker 34 listening for tasks. I0506 03:04:07.004020 1 tasks_processing.go:69] worker 61 listening for tasks. I0506 03:04:07.003969 1 tasks_processing.go:69] worker 63 listening for tasks. I0506 03:04:07.003852 1 tasks_processing.go:69] worker 26 listening for tasks. I0506 03:04:07.003969 1 tasks_processing.go:69] worker 35 listening for tasks. I0506 03:04:07.004043 1 tasks_processing.go:71] worker 5 working on jaegers task. I0506 03:04:07.004045 1 tasks_processing.go:71] worker 1 working on nodenetworkconfigurationpolicies task. I0506 03:04:07.004049 1 tasks_processing.go:71] worker 6 working on monitoring_persistent_volumes task. I0506 03:04:07.004050 1 tasks_processing.go:71] worker 15 working on infrastructures task. I0506 03:04:07.004049 1 tasks_processing.go:71] worker 4 working on feature_gates task. I0506 03:04:07.004059 1 tasks_processing.go:71] worker 36 working on openstack_controlplanes task. I0506 03:04:07.004061 1 tasks_processing.go:71] worker 45 working on validating_webhook_configurations task. I0506 03:04:07.004065 1 tasks_processing.go:71] worker 23 working on machine_sets task. I0506 03:04:07.004064 1 tasks_processing.go:71] worker 30 working on mutating_webhook_configurations task. I0506 03:04:07.004073 1 tasks_processing.go:71] worker 56 working on service_accounts task. I0506 03:04:07.004102 1 tasks_processing.go:71] worker 25 working on ceph_cluster task. I0506 03:04:07.004048 1 tasks_processing.go:71] worker 57 working on oauths task. I0506 03:04:07.004126 1 tasks_processing.go:71] worker 24 working on node_logs task. I0506 03:04:07.004128 1 tasks_processing.go:71] worker 51 working on machine_autoscalers task. I0506 03:04:07.004161 1 tasks_processing.go:71] worker 42 working on pod_network_connectivity_checks task. I0506 03:04:07.004197 1 tasks_processing.go:71] worker 38 working on olm_operators task. I0506 03:04:07.004229 1 tasks_processing.go:71] worker 31 working on sap_config task. I0506 03:04:07.004043 1 tasks_processing.go:71] worker 16 working on overlapping_namespace_uids task. I0506 03:04:07.004315 1 tasks_processing.go:71] worker 37 working on silenced_alerts task. I0506 03:04:07.004334 1 tasks_processing.go:71] worker 28 working on operators_pods_and_events task. I0506 03:04:07.004341 1 tasks_processing.go:71] worker 59 working on machine_healthchecks task. W0506 03:04:07.004345 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:07.004351 1 tasks_processing.go:71] worker 12 working on tsdb_status task. I0506 03:04:07.004357 1 tasks_processing.go:71] worker 37 working on aggregated_monitoring_cr_names task. W0506 03:04:07.004374 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:07.004389 1 tasks_processing.go:71] worker 44 working on machine_configs task. I0506 03:04:07.004393 1 tasks_processing.go:71] worker 48 working on config_maps task. I0506 03:04:07.004398 1 tasks_processing.go:71] worker 49 working on pdbs task. I0506 03:04:07.004416 1 tasks_processing.go:71] worker 41 working on support_secret task. I0506 03:04:07.004491 1 tasks_processing.go:71] worker 54 working on crds task. I0506 03:04:07.004591 1 tasks_processing.go:71] worker 3 working on openshift_machine_api_events task. I0506 03:04:07.004616 1 tasks_processing.go:71] worker 47 working on certificate_signing_requests task. I0506 03:04:07.004642 1 tasks_processing.go:71] worker 11 working on proxies task. I0506 03:04:07.004684 1 tasks_processing.go:71] worker 9 working on sap_pods task. I0506 03:04:07.004740 1 tasks_processing.go:71] worker 43 working on openshift_logging task. I0506 03:04:07.004790 1 tasks_processing.go:71] worker 58 working on openstack_version task. I0506 03:04:07.004395 1 tasks_processing.go:71] worker 50 working on operators task. I0506 03:04:07.004054 1 tasks_processing.go:71] worker 17 working on install_plans task. I0506 03:04:07.004908 1 tasks_processing.go:71] worker 22 working on version task. I0506 03:04:07.004054 1 tasks_processing.go:71] worker 46 working on lokistack task. I0506 03:04:07.005079 1 tasks_processing.go:71] worker 8 working on image_pruners task. I0506 03:04:07.004058 1 tasks_processing.go:71] worker 2 working on image_registries task. I0506 03:04:07.004122 1 tasks_processing.go:71] worker 39 working on networks task. I0506 03:04:07.004123 1 tasks_processing.go:71] worker 21 working on container_runtime_configs task. I0506 03:04:07.005101 1 tasks_processing.go:71] worker 53 working on container_images task. I0506 03:04:07.005316 1 tasks_processing.go:71] worker 20 working on active_alerts task. I0506 03:04:07.004057 1 tasks_processing.go:71] worker 40 working on ingress_certificates task. W0506 03:04:07.005358 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:07.005376 1 tasks_processing.go:71] worker 62 working on schedulers task. I0506 03:04:07.005473 1 tasks_processing.go:71] worker 34 working on storage_cluster task. I0506 03:04:07.004060 1 tasks_processing.go:71] worker 52 working on authentication task. I0506 03:04:07.005484 1 tasks_processing.go:71] worker 61 working on ingress task. I0506 03:04:07.005489 1 tasks_processing.go:71] worker 63 working on storage_classes task. I0506 03:04:07.005558 1 tasks_processing.go:71] worker 33 working on nodenetworkstates task. I0506 03:04:07.005566 1 tasks_processing.go:71] worker 29 working on dvo_metrics task. I0506 03:04:07.004051 1 tasks_processing.go:71] worker 35 working on machines task. I0506 03:04:07.005712 1 tasks_processing.go:71] worker 18 working on metrics task. I0506 03:04:07.005719 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 26.172µs to process 0 records I0506 03:04:07.005733 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 29.591µs to process 0 records I0506 03:04:07.005742 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 37.11µs to process 0 records W0506 03:04:07.005740 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:07.005750 1 tasks_processing.go:71] worker 60 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0506 03:04:07.005751 1 tasks_processing.go:71] worker 10 working on qemu_kubevirt_launcher_logs task. I0506 03:04:07.005775 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 47.59µs to process 0 records I0506 03:04:07.005826 1 tasks_processing.go:74] worker 20 stopped. I0506 03:04:07.005784 1 tasks_processing.go:74] worker 18 stopped. I0506 03:04:07.005794 1 tasks_processing.go:74] worker 12 stopped. I0506 03:04:07.005805 1 tasks_processing.go:71] worker 27 working on machine_config_pools task. I0506 03:04:07.005890 1 tasks_processing.go:71] worker 13 working on nodes task. I0506 03:04:07.005808 1 tasks_processing.go:71] worker 19 working on openstack_dataplanenodesets task. I0506 03:04:07.005817 1 tasks_processing.go:71] worker 26 working on sap_datahubs task. I0506 03:04:07.005832 1 tasks_processing.go:71] worker 55 working on clusterroles task. I0506 03:04:07.005873 1 tasks_processing.go:71] worker 32 working on openstack_dataplanedeployments task. I0506 03:04:07.007591 1 tasks_processing.go:74] worker 7 stopped. I0506 03:04:07.007616 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 3.714987ms to process 0 records I0506 03:04:07.008009 1 tasks_processing.go:74] worker 5 stopped. I0506 03:04:07.008041 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 3.955044ms to process 0 records I0506 03:04:07.008341 1 tasks_processing.go:74] worker 23 stopped. I0506 03:04:07.008354 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 4.267682ms to process 0 records I0506 03:04:07.009111 1 tasks_processing.go:74] worker 36 stopped. I0506 03:04:07.009130 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 5.041161ms to process 0 records I0506 03:04:07.009172 1 tasks_processing.go:74] worker 51 stopped. I0506 03:04:07.009182 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 5.036209ms to process 0 records I0506 03:04:07.009608 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0506 03:04:07.009625 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0506 03:04:07.009630 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0506 03:04:07.009633 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0506 03:04:07.009648 1 controller.go:489] The operator is still being initialized I0506 03:04:07.009659 1 controller.go:512] The operator is healthy I0506 03:04:07.009972 1 tasks_processing.go:74] worker 14 stopped. I0506 03:04:07.010060 1 recorder.go:75] Recording config/image with fingerprint=0cc742fb3a80ac5b43980144f2f68c06adea650b476cb81056cc4f080f108b7e I0506 03:04:07.010072 1 gather.go:177] gatherer "clusterconfig" function "image" took 6.113137ms to process 1 records I0506 03:04:07.010128 1 tasks_processing.go:74] worker 25 stopped. I0506 03:04:07.010142 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 6.006891ms to process 0 records I0506 03:04:07.011539 1 tasks_processing.go:74] worker 42 stopped. E0506 03:04:07.011551 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0506 03:04:07.011559 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 7.361861ms to process 0 records I0506 03:04:07.012965 1 tasks_processing.go:74] worker 0 stopped. I0506 03:04:07.013129 1 recorder.go:75] Recording config/apiserver with fingerprint=81da3f84671f505fbf45aec739c2ead0e56dfe4542d8150500789bb09bf0d613 I0506 03:04:07.013143 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 8.987497ms to process 1 records I0506 03:04:07.013501 1 tasks_processing.go:74] worker 46 stopped. I0506 03:04:07.013511 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 8.505104ms to process 0 records I0506 03:04:07.014573 1 tasks_processing.go:74] worker 43 stopped. I0506 03:04:07.014585 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 9.81839ms to process 0 records I0506 03:04:07.015187 1 tasks_processing.go:74] worker 1 stopped. I0506 03:04:07.015193 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 11.128859ms to process 0 records I0506 03:04:07.029969 1 tasks_processing.go:74] worker 6 stopped. I0506 03:04:07.029982 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 25.908624ms to process 0 records I0506 03:04:07.030451 1 tasks_processing.go:74] worker 31 stopped. I0506 03:04:07.030462 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 26.211463ms to process 0 records I0506 03:04:07.030588 1 tasks_processing.go:74] worker 21 stopped. I0506 03:04:07.030606 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 24.700854ms to process 0 records I0506 03:04:07.030666 1 tasks_processing.go:74] worker 59 stopped. E0506 03:04:07.030675 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0506 03:04:07.030684 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 26.311973ms to process 0 records E0506 03:04:07.030692 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0506 03:04:07.030697 1 gather.go:177] gatherer "clusterconfig" function "machines" took 25.038296ms to process 0 records I0506 03:04:07.030703 1 tasks_processing.go:74] worker 35 stopped. I0506 03:04:07.031619 1 tasks_processing.go:74] worker 58 stopped. I0506 03:04:07.031628 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 26.819237ms to process 0 records I0506 03:04:07.031632 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 26.052435ms to process 0 records I0506 03:04:07.031636 1 tasks_processing.go:74] worker 33 stopped. I0506 03:04:07.032525 1 tasks_processing.go:74] worker 15 stopped. I0506 03:04:07.033343 1 recorder.go:75] Recording config/infrastructure with fingerprint=35c3d22b5fb4f0ed7aff0743ac3feddf19372773148732c818ed59c1b67226cc I0506 03:04:07.033369 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 28.459483ms to process 1 records I0506 03:04:07.033475 1 tasks_processing.go:74] worker 4 stopped. I0506 03:04:07.033543 1 recorder.go:75] Recording config/featuregate with fingerprint=01dcd1639c54cf7bf80d9f3b1085bca942759f72b821b0d40afb4943df412089 I0506 03:04:07.033558 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 28.516049ms to process 1 records I0506 03:04:07.033647 1 tasks_processing.go:74] worker 52 stopped. I0506 03:04:07.033786 1 recorder.go:75] Recording config/authentication with fingerprint=a2fe4c24b27ff55ce0ac9ffd6aa77fb19591e6cf112450e6449caf12682628e7 I0506 03:04:07.033797 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 26.896046ms to process 1 records I0506 03:04:07.033805 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 28.6157ms to process 0 records I0506 03:04:07.033882 1 tasks_processing.go:74] worker 9 stopped. I0506 03:04:07.033958 1 tasks_processing.go:74] worker 57 stopped. I0506 03:04:07.034210 1 recorder.go:75] Recording config/oauth with fingerprint=e344fea9a13472ffe890c4e86709bdd5db1fda3bc1b27df30f9c7906b166acf5 I0506 03:04:07.034223 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 29.199038ms to process 1 records I0506 03:04:07.034237 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 27.293388ms to process 0 records I0506 03:04:07.034244 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 26.390099ms to process 0 records I0506 03:04:07.034271 1 tasks_processing.go:74] worker 19 stopped. I0506 03:04:07.034282 1 tasks_processing.go:74] worker 32 stopped. I0506 03:04:07.034343 1 tasks_processing.go:74] worker 49 stopped. I0506 03:04:07.034703 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=a954526e712ba5fb69d434e3f7234b0232f3e38b1ef402855afd868c58312193 I0506 03:04:07.034741 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=09fdbf360c3e6077799fd01c7af1e081279d4e242f5d461fb29dd5f9e0dc1831 I0506 03:04:07.034766 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=f2d278b15c26b99bf8777fa717303995eacfdd3976a849add0810cb8b5ca1b86 I0506 03:04:07.034780 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.060525ms to process 3 records I0506 03:04:07.034921 1 tasks_processing.go:74] worker 61 stopped. I0506 03:04:07.034956 1 recorder.go:75] Recording config/ingress with fingerprint=cb1cd6a8cffaa0f490fe7a42d1c4940d8460a30a80628530f7bb07739434f03e I0506 03:04:07.034966 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 28.310425ms to process 1 records I0506 03:04:07.034976 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 29.288522ms to process 0 records I0506 03:04:07.035051 1 tasks_processing.go:74] worker 3 stopped. I0506 03:04:07.035118 1 tasks_processing.go:74] worker 45 stopped. I0506 03:04:07.035389 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=9dbcb87da52465789fc8b5c638db71589ee9756f13ca7014ece354d6e9ab1cf5 I0506 03:04:07.035784 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=f974cab1c4fa9423f7a51bb9b0cfd20e0ab24251c83eb6feca6b520fc99c1a6f I0506 03:04:07.036302 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=e15a79f4c07d8bd388387dad9742e7606a4cb4c517e5f62e3da2565b817a5e58 I0506 03:04:07.036423 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=71cd7500ed8ce764bce41c4fd252b0f1fec0ce747847b5be71911cbe1484ed17 I0506 03:04:07.036514 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=9061f2033f5093130330eca1c9b7b995725d7847c4ce9593f6e6574f93800e4d I0506 03:04:07.036594 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=ebfa95bc6ee08fa1b246c400ded6310a70020606be70d7ea0f65db868b6b04a2 I0506 03:04:07.036672 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=c42ee231782ed22eb46fe14447ee851e26ff092d9462b0a52fce1f6d3b599917 I0506 03:04:07.036802 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=f7df4b40b1ef821a84c222a460b7b1770e4dc44825638123b1e510204bfa797a I0506 03:04:07.036862 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=0ad6016c36fe342744b88baf7011dafc7eb78d1a7256f25eb571847276a74d03 I0506 03:04:07.036894 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=5d733521df0c74042835cc18c891498b80dc8fc428994d0b6b370bd1fc6b007b I0506 03:04:07.036931 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=6d3c9a57bd0b59ef5f4e35cd93877829d0a12958c7205206330ae0ccf1b5b463 I0506 03:04:07.036940 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.883117ms to process 11 records I0506 03:04:07.036950 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 30.639274ms to process 0 records I0506 03:04:07.036968 1 tasks_processing.go:74] worker 47 stopped. I0506 03:04:07.037233 1 tasks_processing.go:74] worker 62 stopped. I0506 03:04:07.037429 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=fb90c715b820a70e8ba3ef647a095b6797f5c3d6178306ce1bde5786e4769a1e I0506 03:04:07.037731 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 31.846685ms to process 1 records I0506 03:04:07.038864 1 tasks_processing.go:74] worker 2 stopped. I0506 03:04:07.039599 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=b3bbf33e778c914fb532a04efa63bb7a3780fcc7ab8ae6265088507bb1f19849 I0506 03:04:07.039618 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 33.010798ms to process 1 records I0506 03:04:07.043997 1 gather_logs.go:145] no pods in namespace were found I0506 03:04:07.044009 1 tasks_processing.go:74] worker 10 stopped. I0506 03:04:07.044022 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 38.245676ms to process 0 records I0506 03:04:07.044066 1 tasks_processing.go:74] worker 26 stopped. I0506 03:04:07.044088 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 37.501084ms to process 0 records I0506 03:04:07.044173 1 tasks_processing.go:74] worker 34 stopped. I0506 03:04:07.044184 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 38.472145ms to process 0 records I0506 03:04:07.044338 1 tasks_processing.go:74] worker 11 stopped. I0506 03:04:07.044413 1 recorder.go:75] Recording config/proxy with fingerprint=144735bd5274dafda659826d0ffc08fea7519d545e8e951377950f826ca9de9b I0506 03:04:07.044430 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 39.684513ms to process 1 records I0506 03:04:07.044506 1 tasks_processing.go:74] worker 39 stopped. I0506 03:04:07.044561 1 recorder.go:75] Recording config/network with fingerprint=64f1197e0575e5f8fba115d7759d0619776703666ab9e60bf8a8405bb9c6ed72 I0506 03:04:07.044571 1 gather.go:177] gatherer "clusterconfig" function "networks" took 38.507109ms to process 1 records E0506 03:04:07.044577 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0506 03:04:07.044582 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 40.043091ms to process 0 records I0506 03:04:07.044590 1 tasks_processing.go:74] worker 41 stopped. I0506 03:04:07.044698 1 tasks_processing.go:74] worker 24 stopped. I0506 03:04:07.044707 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 40.566213ms to process 0 records I0506 03:04:07.044741 1 tasks_processing.go:74] worker 63 stopped. I0506 03:04:07.044842 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7a970730dcf9cc2761837ee8916651795208d013893135c6e6867e3ded5a09d7 I0506 03:04:07.044857 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=43b24f0032dbed8227c33bc57f30f39ee88ea87f511a406119d8586d3f9a2a58 I0506 03:04:07.044864 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 39.244222ms to process 2 records I0506 03:04:07.045521 1 tasks_processing.go:74] worker 13 stopped. I0506 03:04:07.045602 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0506 03:04:07.045643 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0506 03:04:07.045856 1 operator.go:288] started I0506 03:04:07.045876 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0506 03:04:07.046013 1 recorder.go:75] Recording config/node/ip-10-0-1-59.ec2.internal with fingerprint=a74cf741b363bcca4b7cf63ef6e9e866fc3fbbf9f8032dc8bf8edf6300bc07e2 I0506 03:04:07.046036 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 39.619299ms to process 1 records I0506 03:04:07.046119 1 tasks_processing.go:74] worker 30 stopped. I0506 03:04:07.046191 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=b0d24d3ca8bc24e45104ce3cb9f08f028685530b5edefc721d8c038584d917bd I0506 03:04:07.046284 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=5228464ff6ec45aac469a2316f3ebe8e9b248f0dbc082d76c437f30aaa0eab74 I0506 03:04:07.046330 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=bcc92121faafdd4cab1de3da46cdca3dd692b92da30acc9340bf6d8ff1bce2fc I0506 03:04:07.046341 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 41.559948ms to process 3 records I0506 03:04:07.046398 1 recorder.go:75] Recording config/olm_operators with fingerprint=8d7fe922beefeacde996e3cfc687e8f51fcbb3866d4ee7406f3d1d0ec30cc613 I0506 03:04:07.046406 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 41.543344ms to process 1 records I0506 03:04:07.046412 1 tasks_processing.go:74] worker 38 stopped. I0506 03:04:07.047090 1 tasks_processing.go:74] worker 16 stopped. I0506 03:04:07.047110 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0506 03:04:07.047122 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 42.812017ms to process 1 records I0506 03:04:07.049459 1 tasks_processing.go:74] worker 54 stopped. I0506 03:04:07.050680 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=3219da47dda2efc8cf40555e0cc4e076c5826baf067aa28d8ced8f693459304a I0506 03:04:07.050880 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=211750ba553f2d45eab73c30e66eb22c7f0619e1ef7e90ba5bae0f81291d274c I0506 03:04:07.050891 1 gather.go:177] gatherer "clusterconfig" function "crds" took 44.936793ms to process 2 records W0506 03:04:07.051430 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 03:04:07.051590 1 tasks_processing.go:74] worker 53 stopped. I0506 03:04:07.052881 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-twgts with fingerprint=27f1df7d6a0312f076f93c966875d3555cd0b781f29f345ae6346fca86f2ded2 I0506 03:04:07.052939 1 recorder.go:75] Recording config/running_containers with fingerprint=b438f86c7bf1a20bd69fbed1e5c61d1f8c5077f15fa636dadfcdf7936bb89867 I0506 03:04:07.052949 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 46.398511ms to process 2 records I0506 03:04:07.054852 1 controller.go:212] Source scaController *sca.Controller is not ready I0506 03:04:07.054863 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0506 03:04:07.054866 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0506 03:04:07.054869 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0506 03:04:07.054872 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0506 03:04:07.054889 1 controller.go:489] The operator is still being initialized I0506 03:04:07.054897 1 controller.go:512] The operator is healthy I0506 03:04:07.061129 1 tasks_processing.go:74] worker 55 stopped. I0506 03:04:07.061305 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=39de4fc9faed2c940c620dc09d3521a683219bac9a6952fbd8f09ae66e75ff7f I0506 03:04:07.061389 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=7410d39521b42745d16f681e45851eca28c4a85dc0c38cef1685fad299e3070a I0506 03:04:07.061398 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 54.324311ms to process 2 records I0506 03:04:07.061858 1 tasks_processing.go:74] worker 60 stopped. I0506 03:04:07.061874 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 56.092623ms to process 0 records E0506 03:04:07.062991 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27485a3459-fad0-4cb8-a022-6b659e97d9c2%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40440->172.30.0.10:53: read: connection refused I0506 03:04:07.063006 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27485a3459-fad0-4cb8-a022-6b659e97d9c2%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40440->172.30.0.10:53: read: connection refused I0506 03:04:07.069749 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0506 03:04:07.069817 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0506 03:04:07.069890 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0506 03:04:07.070552 1 prometheus_rules.go:88] Prometheus rules successfully created I0506 03:04:07.071563 1 tasks_processing.go:74] worker 48 stopped. E0506 03:04:07.071576 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0506 03:04:07.071582 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0506 03:04:07.071587 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0506 03:04:07.071599 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0506 03:04:07.071624 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0506 03:04:07.071629 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0506 03:04:07.071633 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0506 03:04:07.071637 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0506 03:04:07.071681 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0506 03:04:07.071690 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0506 03:04:07.071695 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 67.154826ms to process 7 records I0506 03:04:07.074515 1 base_controller.go:82] Caches are synced for ConfigController I0506 03:04:07.074525 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0506 03:04:07.082433 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 03:04:07.129804 1 tasks_processing.go:74] worker 22 stopped. I0506 03:04:07.130090 1 recorder.go:75] Recording config/version with fingerprint=f6c9359528102299f9699341bd07bed93f962cd58c384a89a2a5db77b7d8ce96 I0506 03:04:07.130104 1 recorder.go:75] Recording config/id with fingerprint=228f4a33c6c02b267cd91111b7f20b03e21405507e6b86c41079804ebf1af732 I0506 03:04:07.130110 1 gather.go:177] gatherer "clusterconfig" function "version" took 124.872463ms to process 2 records I0506 03:04:07.134500 1 tasks_processing.go:74] worker 44 stopped. I0506 03:04:07.134531 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0506 03:04:07.134543 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 130.094952ms to process 1 records I0506 03:04:07.141111 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0506 03:04:07.144715 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:56536->172.30.0.10:53: read: connection refused I0506 03:04:07.144727 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:56536->172.30.0.10:53: read: connection refused I0506 03:04:07.145229 1 tasks_processing.go:74] worker 8 stopped. I0506 03:04:07.145334 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=512f724d3f16d00a4b12259a8a7052d6f04a52af8d39602e1d29e1aa2cc36d66 I0506 03:04:07.145352 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 139.698204ms to process 1 records I0506 03:04:07.155334 1 tasks_processing.go:74] worker 27 stopped. I0506 03:04:07.155350 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 149.434459ms to process 0 records I0506 03:04:07.163006 1 tasks_processing.go:74] worker 40 stopped. E0506 03:04:07.163025 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0506 03:04:07.163033 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q45oqblirj1dttrkl2v26ci3lb7781m-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q45oqblirj1dttrkl2v26ci3lb7781m-primary-cert-bundle-secret" not found I0506 03:04:07.163120 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=bc605534fb56af14ecfbf1bf0f8452c13284b31d153d44c3f264861ed4d9f92e I0506 03:04:07.163137 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 157.63717ms to process 1 records I0506 03:04:07.246476 1 base_controller.go:82] Caches are synced for LoggingSyncer I0506 03:04:07.246492 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0506 03:04:07.452848 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0506 03:04:07.452863 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0506 03:04:07.454064 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-jxplt pod in namespace openshift-dns (previous: false). I0506 03:04:07.679837 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-jxplt pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-jxplt\" is waiting to start: ContainerCreating" I0506 03:04:07.679854 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-jxplt\" is waiting to start: ContainerCreating" I0506 03:04:07.679863 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-jxplt pod in namespace openshift-dns (previous: false). I0506 03:04:07.851830 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-jxplt pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-jxplt\" is waiting to start: ContainerCreating" I0506 03:04:07.851849 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-jxplt\" is waiting to start: ContainerCreating" I0506 03:04:07.851859 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-bv989 pod in namespace openshift-dns (previous: false). I0506 03:04:08.049647 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 03:04:08.049719 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5889b65678-b49dd pod in namespace openshift-image-registry (previous: false). W0506 03:04:08.049804 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 03:04:08.218645 1 tasks_processing.go:74] worker 37 stopped. I0506 03:04:08.218665 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 1.214272306s to process 0 records I0506 03:04:08.243594 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 03:04:08.243647 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-657ccffc89-btthb pod in namespace openshift-image-registry (previous: false). I0506 03:04:08.448884 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-657ccffc89-btthb pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-657ccffc89-btthb\" is waiting to start: ContainerCreating" I0506 03:04:08.448901 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-657ccffc89-btthb\" is waiting to start: ContainerCreating" I0506 03:04:08.448976 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-657ccffc89-tl8gh pod in namespace openshift-image-registry (previous: false). I0506 03:04:08.485796 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0506 03:04:08.643990 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 03:04:08.644007 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-f4z2h pod in namespace openshift-image-registry (previous: false). I0506 03:04:08.848177 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 03:04:08.848195 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-577f6d9bbc-7f888 pod in namespace openshift-ingress (previous: false). I0506 03:04:09.043887 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 03:04:09.043905 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-577f6d9bbc-r97mn pod in namespace openshift-ingress (previous: false). W0506 03:04:09.049177 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 03:04:09.072937 1 tasks_processing.go:74] worker 50 stopped. I0506 03:04:09.073044 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=0044260e0f03b7e03c610b0ba70cb239d9d43bf6f617f8ef4287fe1e12e0ae4e I0506 03:04:09.073091 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=53fa693d3e8475cc1ba05fe1324df25f3cb51bba186f88e7fd5a6ba10be4f5eb I0506 03:04:09.073148 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0506 03:04:09.073200 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=8eac48246449694c58fab81b5cd2a12d14c09ae4c7171fe86f81a4a38cd8b2dd I0506 03:04:09.073227 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0506 03:04:09.073287 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=65e458ddebd78e924796995452ede15c96145805dff4238c415a8a8524abd3af I0506 03:04:09.073356 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=c62fe4093b93194808f4d7e3cf84577ba5d6fb247593098486d855732761fd8c I0506 03:04:09.073394 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=ad82c5fd0371c31b0ad8054c5a6fdaf9804091047b15c65869697c85bae42ac6 I0506 03:04:09.073416 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=4fe21c05b250d14183169e4b4f8f2987ecc06d58f13550db92c492b1d4f0089d I0506 03:04:09.073445 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=11eb2e615d0a37294c0409ce2870ed22668f0a06aa8ee80d3606df141393c7a5 I0506 03:04:09.073502 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0506 03:04:09.073534 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=7b0091cac72db4b0e5d5b6754ad6d5c9e7de3bdf3bc0d05e6593b4185a2f9297 I0506 03:04:09.073546 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0506 03:04:09.073568 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=d090640b6ed7c34cc3273a10a802cd968296c8067355b0bcda27182a72be4d90 I0506 03:04:09.073581 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0506 03:04:09.073599 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=27908ea0715a4b7577f1d82db3809783fb68b321e000e2d32f8f0f50afb9bdf7 I0506 03:04:09.073611 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0506 03:04:09.073634 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=e0db5dc57a05882c9b55c24095adb052cb7c04761b4da7d559a1f5a4f9c12622 I0506 03:04:09.073811 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=490491b512249f3ec78e3ec098e5492b44ad29cad4266373cf6cf87a9a0be5cd I0506 03:04:09.073853 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0506 03:04:09.073869 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0506 03:04:09.073899 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0506 03:04:09.073958 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=8f7dd35176a0b89db4706be4417c7418772979a9dc41370ce26bc86afc6ecb71 I0506 03:04:09.073993 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=1d6c08d745203e229fe7cfa829f589d3ccfea028e8293f707e587c487e9de6b2 I0506 03:04:09.074003 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0506 03:04:09.074026 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=3cd3b0a7fb66e53304d8fa8042ca7a7194893da7f65a7d9d24649c7604258e5a I0506 03:04:09.074036 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0506 03:04:09.074055 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=32bacc87244d6f07074e81a9550bafb24091fee0052bd81656b8866a33a52e44 I0506 03:04:09.074078 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=1f5bb50864c32cd6528d118fdc3195bdfd69f04272dd8ce6ce90f3b5c1def31a I0506 03:04:09.074102 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=5cb5d415ee30b3bfbfe1ecbbbea6e029b7972ff0f7181d45a6074cb2f085632c I0506 03:04:09.074121 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=4e01a00cc2a64c3909a5ba79ab8ac3ab4134351abf51bc1a5b28836cc500f1d3 I0506 03:04:09.074178 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=ca9fb33f3815f1e63e2284cf816cb76f1b2828076c15ce3d357d00a49df3b430 I0506 03:04:09.074192 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0506 03:04:09.074228 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=f2adc7811c62dbc1f655916e0cc5e07bd34fd9045479078278902b978878eaa7 I0506 03:04:09.074270 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0506 03:04:09.074315 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0506 03:04:09.074329 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.068114659s to process 36 records I0506 03:04:09.258531 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-577f6d9bbc-r97mn pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-577f6d9bbc-r97mn\" is waiting to start: ContainerCreating" I0506 03:04:09.258547 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-577f6d9bbc-r97mn\" is waiting to start: ContainerCreating" I0506 03:04:09.258559 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-685c994f87-m2t8m pod in namespace openshift-ingress (previous: false). I0506 03:04:09.449047 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-685c994f87-m2t8m pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-685c994f87-m2t8m\" is waiting to start: ContainerCreating" I0506 03:04:09.449063 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-685c994f87-m2t8m\" is waiting to start: ContainerCreating" I0506 03:04:09.449094 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-7xn5j pod in namespace openshift-ingress-canary (previous: false). I0506 03:04:09.649053 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-7xn5j pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-7xn5j\" is waiting to start: ContainerCreating" I0506 03:04:09.649072 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-7xn5j\" is waiting to start: ContainerCreating" I0506 03:04:09.649090 1 tasks_processing.go:74] worker 28 stopped. I0506 03:04:09.649174 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=a3f23ef09072539e37934bc5d7e589d7a67bff128d393f4471bfa18b3bf2efb2 I0506 03:04:09.649202 1 recorder.go:75] Recording events/openshift-dns with fingerprint=92730155b3f1a3775f10607786b7eb189fdbfd5af90e484ad7d49676a77f6dc3 I0506 03:04:09.649291 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=a691531407b864d60f5fd8aa7d197ba7d70490a6bdc39c7ace5276f78c32e726 I0506 03:04:09.649324 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=5ce2358d3e862edb5583c23772e3e045707fba0a5137d7ccb446bb68fcf0d87c I0506 03:04:09.649381 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=d1e21839f7169f5c6822489204857c30ba1ee35e2c223ab3a88056d6ee8ace78 I0506 03:04:09.649394 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=da9d5cc80f0b633126d9422df1048e8e60a78a70592c336b9e32119f3cf8c4d2 I0506 03:04:09.649552 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-jxplt with fingerprint=0f9939d2b9a14fc46030bbce2ab4f8b4dcea9d1c27eed8c2cb820d7fa38af30e I0506 03:04:09.649669 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5889b65678-b49dd with fingerprint=ba66d2778785d95d13da6103dff8569cb7eb688af1ed24abd43d77e47d6e7f72 I0506 03:04:09.649796 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-657ccffc89-btthb with fingerprint=f61d879206b63480cb5ec8e80e647c99090ac7ae0b173f8b895d354a5273a205 I0506 03:04:09.649887 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-657ccffc89-tl8gh with fingerprint=fc28556034babf759e1c181b9aec00c7bcec0a2d794587e25be445e24f2874b3 I0506 03:04:09.649957 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-7xn5j with fingerprint=ccad2d6f41ade13e1abc40357d7ab0fcb659428bacfe25b96336f2ae9ac057b7 I0506 03:04:09.649972 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.644741667s to process 11 records W0506 03:04:10.049754 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0506 03:04:11.050167 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0506 03:04:12.046286 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0506 03:04:12.046323 1 tasks_processing.go:74] worker 29 stopped. E0506 03:04:12.046333 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0506 03:04:12.046344 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0506 03:04:12.046358 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0506 03:04:12.046368 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.040738346s to process 1 records I0506 03:04:19.444437 1 tasks_processing.go:74] worker 17 stopped. I0506 03:04:19.444479 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0506 03:04:19.444490 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.439542582s to process 1 records I0506 03:04:19.948504 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 03:04:20.120948 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 03:04:20.151120 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 03:04:20.210581 1 tasks_processing.go:74] worker 56 stopped. I0506 03:04:20.210854 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=b3a2c4fa45ed7ae58b9c4093be9a56682fdfea23d7350a26feed0cd7c1d1f420 I0506 03:04:20.210872 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.206487108s to process 1 records E0506 03:04:20.210940 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.207s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0506 03:04:20.212050 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0506 03:04:20.212066 1 periodic.go:209] Running workloads gatherer I0506 03:04:20.212086 1 tasks_processing.go:45] number of workers: 2 I0506 03:04:20.212101 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 03:04:20.212107 1 tasks_processing.go:71] worker 1 working on workload_info task. I0506 03:04:20.212112 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 03:04:20.212128 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0506 03:04:20.231939 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0506 03:04:20.241536 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0506 03:04:20.242045 1 tasks_processing.go:74] worker 0 stopped. I0506 03:04:20.242056 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.90306ms to process 0 records I0506 03:04:20.252384 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (11ms) I0506 03:04:20.262724 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (10ms) I0506 03:04:20.271143 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (8ms) I0506 03:04:20.279959 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (9ms) I0506 03:04:20.290518 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (11ms) I0506 03:04:20.299653 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (9ms) I0506 03:04:20.308491 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (9ms) I0506 03:04:20.317990 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (9ms) I0506 03:04:20.326573 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (9ms) I0506 03:04:20.343213 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (17ms) I0506 03:04:20.441685 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (98ms) I0506 03:04:20.547379 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (106ms) I0506 03:04:20.642564 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (95ms) I0506 03:04:20.742009 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0506 03:04:20.844756 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (103ms) I0506 03:04:20.944714 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0506 03:04:21.041994 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (97ms) I0506 03:04:21.142704 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (101ms) I0506 03:04:21.241399 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (99ms) I0506 03:04:21.346922 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (106ms) I0506 03:04:21.442240 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (95ms) I0506 03:04:21.547000 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (105ms) I0506 03:04:21.642338 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (95ms) I0506 03:04:21.744502 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (102ms) I0506 03:04:21.842067 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (98ms) I0506 03:04:21.946982 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (105ms) I0506 03:04:22.042122 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (95ms) I0506 03:04:22.146869 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (105ms) I0506 03:04:22.242395 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (96ms) I0506 03:04:22.345752 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (103ms) I0506 03:04:22.345783 1 tasks_processing.go:74] worker 1 stopped. E0506 03:04:22.345796 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0506 03:04:22.346111 1 recorder.go:75] Recording config/workload_info with fingerprint=0d66e6329940a6408933f68b84026217337e87fda07efd4a8f888a716ece882c I0506 03:04:22.346129 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.133667183s to process 1 records E0506 03:04:22.346153 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.134s with: function \"workload_info\" failed with an error" I0506 03:04:22.347261 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0506 03:04:22.347273 1 periodic.go:209] Running conditional gatherer I0506 03:04:22.352947 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0506 03:04:22.359484 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:43570->172.30.0.10:53: read: connection refused E0506 03:04:22.359714 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 03:04:22.359768 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0506 03:04:22.365569 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0506 03:04:22.365583 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365588 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365591 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365595 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365598 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365601 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365604 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365607 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 03:04:22.365610 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0506 03:04:22.365624 1 tasks_processing.go:45] number of workers: 3 I0506 03:04:22.365634 1 tasks_processing.go:69] worker 2 listening for tasks. I0506 03:04:22.365638 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0506 03:04:22.365649 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 03:04:22.365658 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 03:04:22.365664 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0506 03:04:22.365666 1 tasks_processing.go:74] worker 1 stopped. I0506 03:04:22.365680 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0506 03:04:22.365737 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0506 03:04:22.365756 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 898ns to process 1 records I0506 03:04:22.365801 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0506 03:04:22.365812 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.219µs to process 1 records I0506 03:04:22.365820 1 tasks_processing.go:74] worker 0 stopped. I0506 03:04:22.365955 1 tasks_processing.go:74] worker 2 stopped. I0506 03:04:22.365965 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 262.053µs to process 0 records I0506 03:04:22.365983 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:43570->172.30.0.10:53: read: connection refused I0506 03:04:22.365999 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0506 03:04:22.389903 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=5fdcf0f471f80b1a783554c1b770348285e8b084435e192d2c6ce313cd217a60 I0506 03:04:22.390032 1 diskrecorder.go:70] Writing 105 records to /var/lib/insights-operator/insights-2026-05-06-030422.tar.gz I0506 03:04:22.396578 1 diskrecorder.go:51] Wrote 105 records to disk in 6ms I0506 03:04:22.396605 1 periodic.go:278] Gathering cluster info every 2h0m0s I0506 03:04:22.396620 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0506 03:04:31.009668 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 03:05:16.580557 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="24f517585c9df59b0a5628ec39cfb54fd28d0d87c5080df5b58994e4c61db97d") W0506 03:05:16.580596 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was created I0506 03:05:16.580669 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0506 03:05:16.580686 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="8851663703442dd8eafa627069951fab06ca9def1af9d6904575ee6f8bea223c") I0506 03:05:16.580689 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0506 03:05:16.580742 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="0936ade334b00abe66d942ecb0e7deb5de6ecd25063b344cee67355598a68967") I0506 03:05:16.580744 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0506 03:05:16.580755 1 periodic.go:170] Shutting down I0506 03:05:16.580759 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0506 03:05:16.580777 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0506 03:05:16.580788 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0506 03:05:16.580793 1 secure_serving.go:258] Stopped listening on [::]:8443 I0506 03:05:16.580788 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0506 03:05:16.580795 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0506 03:05:16.580811 1 base_controller.go:181] Shutting down ConfigController ... I0506 03:05:16.580814 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0506 03:05:16.580812 1 genericapiserver.go:600] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0506 03:05:16.580798 1 base_controller.go:181] Shutting down LoggingSyncer ... I0506 03:05:16.580827 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0506 03:05:16.580831 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0506 03:05:16.580834 1 base_controller.go:113] All ConfigController workers have been terminated I0506 03:05:16.580830 1 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/tmp/serving-cert-1008527116/tls.crt::/tmp/serving-cert-1008527116/tls.key"