W0405 15:04:45.443037 1 cmd.go:257] Using insecure, self-signed certificates I0405 15:04:45.785393 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:04:45.785692 1 observer_polling.go:159] Starting file observer I0405 15:04:46.151610 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0405 15:04:46.151890 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0405 15:04:46.152832 1 secure_serving.go:57] Forcing use of http/1.1 only I0405 15:04:46.152851 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0405 15:04:46.152861 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0405 15:04:46.152899 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0405 15:04:46.152914 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0405 15:04:46.152919 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0405 15:04:46.152923 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0405 15:04:46.152927 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0405 15:04:46.155935 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0405 15:04:46.155944 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"67bc4ef7-622d-4612-ad8d-7d5bf18946f6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0405 15:04:46.156521 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0405 15:04:46.156539 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0405 15:04:46.156541 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0405 15:04:46.156541 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0405 15:04:46.156577 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0405 15:04:46.156582 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0405 15:04:46.156891 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3107701887/tls.crt::/tmp/serving-cert-3107701887/tls.key" I0405 15:04:46.157375 1 secure_serving.go:213] Serving securely on [::]:8443 I0405 15:04:46.157410 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0405 15:04:46.159799 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0405 15:04:46.159822 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0405 15:04:46.159914 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0405 15:04:46.164037 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0405 15:04:46.164052 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0405 15:04:46.167300 1 secretconfigobserver.go:119] support secret does not exist I0405 15:04:46.172727 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0405 15:04:46.181339 1 secretconfigobserver.go:119] support secret does not exist I0405 15:04:46.191638 1 recorder.go:161] Pruning old reports every 8h36m19s, max age is 288h0m0s I0405 15:04:46.208416 1 periodic.go:209] Running clusterconfig gatherer I0405 15:04:46.208429 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0405 15:04:46.208437 1 insightsreport.go:296] Starting report retriever I0405 15:04:46.208443 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0405 15:04:46.208416 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0405 15:04:46.208476 1 tasks_processing.go:45] number of workers: 64 I0405 15:04:46.208490 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0405 15:04:46.208519 1 tasks_processing.go:69] worker 13 listening for tasks. I0405 15:04:46.208529 1 tasks_processing.go:69] worker 7 listening for tasks. I0405 15:04:46.208535 1 tasks_processing.go:69] worker 11 listening for tasks. I0405 15:04:46.208535 1 tasks_processing.go:69] worker 10 listening for tasks. I0405 15:04:46.208535 1 tasks_processing.go:69] worker 6 listening for tasks. I0405 15:04:46.208540 1 tasks_processing.go:69] worker 12 listening for tasks. I0405 15:04:46.208545 1 tasks_processing.go:69] worker 8 listening for tasks. I0405 15:04:46.208546 1 tasks_processing.go:69] worker 4 listening for tasks. I0405 15:04:46.208552 1 tasks_processing.go:71] worker 8 working on install_plans task. I0405 15:04:46.208550 1 tasks_processing.go:69] worker 2 listening for tasks. I0405 15:04:46.208557 1 tasks_processing.go:69] worker 9 listening for tasks. I0405 15:04:46.208545 1 tasks_processing.go:71] worker 6 working on image_registries task. I0405 15:04:46.208566 1 tasks_processing.go:69] worker 15 listening for tasks. I0405 15:04:46.208568 1 tasks_processing.go:71] worker 9 working on nodenetworkconfigurationpolicies task. I0405 15:04:46.208567 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 15:04:46.208573 1 tasks_processing.go:71] worker 15 working on crds task. I0405 15:04:46.208556 1 tasks_processing.go:69] worker 14 listening for tasks. I0405 15:04:46.208570 1 tasks_processing.go:69] worker 3 listening for tasks. I0405 15:04:46.208580 1 tasks_processing.go:69] worker 5 listening for tasks. I0405 15:04:46.208586 1 tasks_processing.go:69] worker 41 listening for tasks. I0405 15:04:46.208563 1 tasks_processing.go:71] worker 2 working on dvo_metrics task. I0405 15:04:46.208590 1 tasks_processing.go:71] worker 12 working on certificate_signing_requests task. I0405 15:04:46.208594 1 tasks_processing.go:69] worker 16 listening for tasks. I0405 15:04:46.208595 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 15:04:46.208601 1 tasks_processing.go:69] worker 17 listening for tasks. I0405 15:04:46.208609 1 tasks_processing.go:69] worker 18 listening for tasks. I0405 15:04:46.208614 1 tasks_processing.go:69] worker 47 listening for tasks. I0405 15:04:46.208609 1 tasks_processing.go:69] worker 54 listening for tasks. I0405 15:04:46.208617 1 tasks_processing.go:69] worker 30 listening for tasks. I0405 15:04:46.208622 1 tasks_processing.go:69] worker 48 listening for tasks. I0405 15:04:46.208626 1 tasks_processing.go:69] worker 31 listening for tasks. I0405 15:04:46.208623 1 tasks_processing.go:69] worker 45 listening for tasks. I0405 15:04:46.208628 1 tasks_processing.go:69] worker 46 listening for tasks. I0405 15:04:46.208635 1 tasks_processing.go:69] worker 32 listening for tasks. I0405 15:04:46.208635 1 tasks_processing.go:69] worker 44 listening for tasks. I0405 15:04:46.208641 1 tasks_processing.go:69] worker 42 listening for tasks. I0405 15:04:46.208645 1 tasks_processing.go:69] worker 33 listening for tasks. I0405 15:04:46.208639 1 tasks_processing.go:69] worker 49 listening for tasks. I0405 15:04:46.208649 1 tasks_processing.go:69] worker 50 listening for tasks. I0405 15:04:46.208654 1 tasks_processing.go:69] worker 43 listening for tasks. I0405 15:04:46.208658 1 tasks_processing.go:69] worker 20 listening for tasks. I0405 15:04:46.208662 1 tasks_processing.go:69] worker 51 listening for tasks. I0405 15:04:46.208657 1 tasks_processing.go:69] worker 34 listening for tasks. I0405 15:04:46.208665 1 tasks_processing.go:69] worker 38 listening for tasks. I0405 15:04:46.208670 1 tasks_processing.go:69] worker 21 listening for tasks. I0405 15:04:46.208673 1 tasks_processing.go:69] worker 55 listening for tasks. I0405 15:04:46.208674 1 tasks_processing.go:69] worker 61 listening for tasks. I0405 15:04:46.208673 1 tasks_processing.go:69] worker 52 listening for tasks. I0405 15:04:46.208553 1 tasks_processing.go:71] worker 4 working on pdbs task. I0405 15:04:46.208682 1 tasks_processing.go:69] worker 53 listening for tasks. I0405 15:04:46.208686 1 tasks_processing.go:69] worker 56 listening for tasks. I0405 15:04:46.208649 1 tasks_processing.go:69] worker 19 listening for tasks. I0405 15:04:46.208690 1 tasks_processing.go:69] worker 22 listening for tasks. I0405 15:04:46.208694 1 tasks_processing.go:69] worker 24 listening for tasks. I0405 15:04:46.208659 1 tasks_processing.go:69] worker 39 listening for tasks. I0405 15:04:46.208699 1 tasks_processing.go:69] worker 62 listening for tasks. I0405 15:04:46.208701 1 tasks_processing.go:69] worker 59 listening for tasks. I0405 15:04:46.208701 1 tasks_processing.go:69] worker 35 listening for tasks. I0405 15:04:46.208706 1 tasks_processing.go:71] worker 11 working on clusterroles task. I0405 15:04:46.208710 1 tasks_processing.go:69] worker 36 listening for tasks. I0405 15:04:46.208699 1 tasks_processing.go:69] worker 28 listening for tasks. I0405 15:04:46.208664 1 tasks_processing.go:69] worker 40 listening for tasks. I0405 15:04:46.208679 1 tasks_processing.go:69] worker 25 listening for tasks. I0405 15:04:46.208546 1 tasks_processing.go:71] worker 10 working on sap_pods task. I0405 15:04:46.208745 1 tasks_processing.go:71] worker 25 working on infrastructures task. I0405 15:04:46.208684 1 tasks_processing.go:69] worker 26 listening for tasks. I0405 15:04:46.208778 1 tasks_processing.go:71] worker 50 working on machine_sets task. I0405 15:04:46.208629 1 tasks_processing.go:69] worker 29 listening for tasks. I0405 15:04:46.208795 1 tasks_processing.go:71] worker 3 working on machines task. I0405 15:04:46.208796 1 tasks_processing.go:71] worker 56 working on proxies task. I0405 15:04:46.208800 1 tasks_processing.go:71] worker 29 working on machine_config_pools task. I0405 15:04:46.208825 1 tasks_processing.go:71] worker 46 working on nodes task. I0405 15:04:46.208866 1 tasks_processing.go:71] worker 30 working on lokistack task. I0405 15:04:46.208938 1 tasks_processing.go:71] worker 43 working on cluster_apiserver task. I0405 15:04:46.208973 1 tasks_processing.go:71] worker 36 working on ingress_certificates task. I0405 15:04:46.209022 1 tasks_processing.go:71] worker 16 working on silenced_alerts task. I0405 15:04:46.209070 1 tasks_processing.go:71] worker 40 working on machine_healthchecks task. I0405 15:04:46.208947 1 tasks_processing.go:71] worker 19 working on service_accounts task. I0405 15:04:46.209084 1 tasks_processing.go:71] worker 31 working on openshift_machine_api_events task. W0405 15:04:46.209094 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:04:46.209107 1 tasks_processing.go:71] worker 16 working on sap_config task. I0405 15:04:46.209114 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 29.496µs to process 0 records I0405 15:04:46.209007 1 tasks_processing.go:71] worker 1 working on tsdb_status task. W0405 15:04:46.209266 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:04:46.209278 1 tasks_processing.go:71] worker 1 working on container_runtime_configs task. I0405 15:04:46.208689 1 tasks_processing.go:69] worker 63 listening for tasks. I0405 15:04:46.208692 1 tasks_processing.go:69] worker 23 listening for tasks. I0405 15:04:46.208694 1 tasks_processing.go:69] worker 58 listening for tasks. I0405 15:04:46.208701 1 tasks_processing.go:71] worker 13 working on operators task. I0405 15:04:46.208666 1 tasks_processing.go:69] worker 57 listening for tasks. I0405 15:04:46.208699 1 tasks_processing.go:71] worker 7 working on version task. I0405 15:04:46.208780 1 tasks_processing.go:71] worker 26 working on aggregated_monitoring_cr_names task. I0405 15:04:46.208786 1 tasks_processing.go:71] worker 54 working on jaegers task. I0405 15:04:46.208790 1 tasks_processing.go:71] worker 0 working on operators_pods_and_events task. I0405 15:04:46.209806 1 tasks_processing.go:69] worker 37 listening for tasks. I0405 15:04:46.209826 1 tasks_processing.go:71] worker 37 working on overlapping_namespace_uids task. I0405 15:04:46.208793 1 tasks_processing.go:71] worker 14 working on openstack_dataplanenodesets task. I0405 15:04:46.208952 1 tasks_processing.go:71] worker 22 working on config_maps task. I0405 15:04:46.208956 1 tasks_processing.go:71] worker 24 working on cost_management_metrics_configs task. I0405 15:04:46.208960 1 tasks_processing.go:71] worker 39 working on nodenetworkstates task. I0405 15:04:46.208963 1 tasks_processing.go:71] worker 62 working on openstack_controlplanes task. I0405 15:04:46.208966 1 tasks_processing.go:71] worker 59 working on ceph_cluster task. I0405 15:04:46.208970 1 tasks_processing.go:71] worker 35 working on ingress task. I0405 15:04:46.209012 1 tasks_processing.go:71] worker 5 working on olm_operators task. I0405 15:04:46.209018 1 tasks_processing.go:71] worker 41 working on monitoring_persistent_volumes task. I0405 15:04:46.208680 1 tasks_processing.go:69] worker 60 listening for tasks. I0405 15:04:46.209021 1 tasks_processing.go:71] worker 32 working on machine_configs task. I0405 15:04:46.209026 1 tasks_processing.go:71] worker 44 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0405 15:04:46.209027 1 tasks_processing.go:71] worker 33 working on mutating_webhook_configurations task. I0405 15:04:46.209030 1 tasks_processing.go:71] worker 21 working on openstack_dataplanedeployments task. I0405 15:04:46.209030 1 tasks_processing.go:71] worker 49 working on networks task. I0405 15:04:46.209036 1 tasks_processing.go:71] worker 34 working on pod_network_connectivity_checks task. I0405 15:04:46.209032 1 tasks_processing.go:71] worker 42 working on machine_autoscalers task. I0405 15:04:46.209035 1 tasks_processing.go:71] worker 20 working on qemu_kubevirt_launcher_logs task. I0405 15:04:46.209040 1 tasks_processing.go:71] worker 51 working on image_pruners task. I0405 15:04:46.209042 1 tasks_processing.go:71] worker 61 working on active_alerts task. W0405 15:04:46.210949 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:04:46.209042 1 tasks_processing.go:71] worker 38 working on metrics task. I0405 15:04:46.209046 1 tasks_processing.go:71] worker 52 working on schedulers task. I0405 15:04:46.209049 1 tasks_processing.go:71] worker 18 working on image task. I0405 15:04:46.209047 1 tasks_processing.go:71] worker 55 working on support_secret task. I0405 15:04:46.209054 1 tasks_processing.go:71] worker 17 working on validating_webhook_configurations task. I0405 15:04:46.209054 1 tasks_processing.go:71] worker 53 working on storage_classes task. I0405 15:04:46.209060 1 tasks_processing.go:71] worker 45 working on container_images task. I0405 15:04:46.209061 1 tasks_processing.go:71] worker 28 working on feature_gates task. I0405 15:04:46.209066 1 tasks_processing.go:71] worker 48 working on sap_datahubs task. I0405 15:04:46.209075 1 tasks_processing.go:71] worker 47 working on oauths task. I0405 15:04:46.208691 1 tasks_processing.go:69] worker 27 listening for tasks. I0405 15:04:46.209340 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 24.6µs to process 0 records I0405 15:04:46.209412 1 tasks_processing.go:71] worker 57 working on openstack_version task. I0405 15:04:46.209417 1 tasks_processing.go:71] worker 63 working on storage_cluster task. I0405 15:04:46.209419 1 tasks_processing.go:71] worker 23 working on openshift_logging task. I0405 15:04:46.209424 1 tasks_processing.go:71] worker 58 working on authentication task. I0405 15:04:46.210496 1 tasks_processing.go:71] worker 60 working on node_logs task. I0405 15:04:46.211060 1 tasks_processing.go:74] worker 27 stopped. I0405 15:04:46.211158 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 33.308µs to process 0 records I0405 15:04:46.211183 1 tasks_processing.go:74] worker 61 stopped. W0405 15:04:46.210977 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:04:46.211270 1 tasks_processing.go:74] worker 38 stopped. I0405 15:04:46.211294 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 301.229µs to process 0 records I0405 15:04:46.213271 1 tasks_processing.go:74] worker 9 stopped. I0405 15:04:46.213283 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 4.694271ms to process 0 records I0405 15:04:46.213305 1 tasks_processing.go:74] worker 30 stopped. I0405 15:04:46.213318 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.382651ms to process 0 records I0405 15:04:46.213329 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 4.518463ms to process 0 records I0405 15:04:46.213338 1 tasks_processing.go:74] worker 50 stopped. I0405 15:04:46.219648 1 tasks_processing.go:74] worker 21 stopped. I0405 15:04:46.219661 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 9.001003ms to process 0 records I0405 15:04:46.219667 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.698973ms to process 0 records I0405 15:04:46.219671 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 10.37819ms to process 0 records I0405 15:04:46.219673 1 tasks_processing.go:74] worker 24 stopped. I0405 15:04:46.219678 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.525045ms to process 0 records I0405 15:04:46.219681 1 tasks_processing.go:74] worker 62 stopped. I0405 15:04:46.219681 1 tasks_processing.go:74] worker 1 stopped. I0405 15:04:46.219685 1 tasks_processing.go:74] worker 16 stopped. I0405 15:04:46.219682 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 10.564468ms to process 0 records I0405 15:04:46.219696 1 tasks_processing.go:74] worker 59 stopped. I0405 15:04:46.219710 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 9.468705ms to process 0 records I0405 15:04:46.219765 1 tasks_processing.go:74] worker 40 stopped. E0405 15:04:46.219776 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0405 15:04:46.219783 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 10.674745ms to process 0 records I0405 15:04:46.220286 1 tasks_processing.go:74] worker 10 stopped. I0405 15:04:46.220290 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 11.548323ms to process 0 records I0405 15:04:46.226253 1 tasks_processing.go:74] worker 39 stopped. I0405 15:04:46.226262 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 16.191561ms to process 0 records E0405 15:04:46.226270 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0405 15:04:46.226277 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 15.507444ms to process 0 records I0405 15:04:46.226282 1 tasks_processing.go:74] worker 34 stopped. I0405 15:04:46.226288 1 tasks_processing.go:74] worker 54 stopped. I0405 15:04:46.226302 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 16.536637ms to process 0 records I0405 15:04:46.226513 1 tasks_processing.go:74] worker 42 stopped. I0405 15:04:46.226546 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 15.490548ms to process 0 records I0405 15:04:46.226577 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 15.482664ms to process 0 records I0405 15:04:46.226586 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 16.68857ms to process 0 records I0405 15:04:46.226592 1 tasks_processing.go:74] worker 14 stopped. I0405 15:04:46.226605 1 tasks_processing.go:74] worker 57 stopped. I0405 15:04:46.226908 1 tasks_processing.go:74] worker 25 stopped. I0405 15:04:46.228497 1 recorder.go:75] Recording config/infrastructure with fingerprint=1bca209831f331cf8e70bfee804823556e4bf3fe53d980d65bf28e9424ca5cc9 I0405 15:04:46.228514 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 18.150339ms to process 1 records I0405 15:04:46.232899 1 tasks_processing.go:74] worker 23 stopped. I0405 15:04:46.232913 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 21.808195ms to process 0 records I0405 15:04:46.232941 1 tasks_processing.go:74] worker 3 stopped. E0405 15:04:46.232955 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0405 15:04:46.232966 1 gather.go:177] gatherer "clusterconfig" function "machines" took 24.137318ms to process 0 records I0405 15:04:46.233641 1 tasks_processing.go:74] worker 63 stopped. I0405 15:04:46.233659 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 22.543587ms to process 0 records I0405 15:04:46.233670 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 24.546083ms to process 0 records I0405 15:04:46.233676 1 tasks_processing.go:74] worker 31 stopped. I0405 15:04:46.233724 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0405 15:04:46.233745 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0405 15:04:46.233754 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0405 15:04:46.233757 1 tasks_processing.go:74] worker 48 stopped. I0405 15:04:46.233759 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0405 15:04:46.233779 1 controller.go:489] The operator is still being initialized I0405 15:04:46.233776 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 22.72191ms to process 0 records I0405 15:04:46.233788 1 controller.go:512] The operator is healthy I0405 15:04:46.233926 1 tasks_processing.go:74] worker 56 stopped. I0405 15:04:46.234013 1 recorder.go:75] Recording config/proxy with fingerprint=85fe10e7d348c3ef5ba41fa671ae4e3c82151fc90ec29be02c722a3e9026165a I0405 15:04:46.234028 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 25.116185ms to process 1 records I0405 15:04:46.234418 1 tasks_processing.go:74] worker 37 stopped. I0405 15:04:46.234449 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0405 15:04:46.234461 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 24.576255ms to process 1 records I0405 15:04:46.234544 1 tasks_processing.go:74] worker 6 stopped. I0405 15:04:46.235150 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=55fffd6ffe2ad75ef2ae9dbe2e052c15605edc39ff451a3b0ea5ca08b4884f2c I0405 15:04:46.235165 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 25.88138ms to process 1 records I0405 15:04:46.236457 1 tasks_processing.go:74] worker 47 stopped. I0405 15:04:46.236660 1 recorder.go:75] Recording config/oauth with fingerprint=b36b5b42c221073d8671cae7858830338f3b82068c5159acb20d4e9160c1ba21 I0405 15:04:46.236671 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 25.450765ms to process 1 records I0405 15:04:46.237433 1 tasks_processing.go:74] worker 43 stopped. I0405 15:04:46.237559 1 recorder.go:75] Recording config/apiserver with fingerprint=864f8453dd2bf533871117fc9f2ca717be6acf4d83ca5fbc3ab45994502de2c9 I0405 15:04:46.237575 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.44548ms to process 1 records I0405 15:04:46.237659 1 tasks_processing.go:74] worker 35 stopped. I0405 15:04:46.237733 1 recorder.go:75] Recording config/ingress with fingerprint=a70d90090cb0f83a32bacbfc4b9339faa2a969ba48180c714cea25a3c6398d8d I0405 15:04:46.237746 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 27.179409ms to process 1 records I0405 15:04:46.244316 1 tasks_processing.go:74] worker 41 stopped. I0405 15:04:46.244329 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 33.833156ms to process 0 records I0405 15:04:46.244691 1 tasks_processing.go:74] worker 4 stopped. I0405 15:04:46.245084 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=3fe8b471f914fbe0eb53ade1979d282f6f59367cf1003cb6ab4b14d0b5554141 I0405 15:04:46.245169 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a8f198ca3b08f12f79b603b7d50954ea4181f5f9e6804a3f1336b926c6e0ad4c I0405 15:04:46.245245 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=dcb1b4f6863d9a1ad9e70dde674811b0cd2af24529188613655b2611c7827b17 I0405 15:04:46.245276 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 35.996471ms to process 3 records E0405 15:04:46.245314 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0405 15:04:46.245343 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 33.720944ms to process 0 records I0405 15:04:46.245506 1 recorder.go:75] Recording config/image with fingerprint=e9d78eb4624fbdb89f1844ca8b25810f79f44c4d96a8209ff6b808d076d85baa I0405 15:04:46.245531 1 tasks_processing.go:74] worker 55 stopped. I0405 15:04:46.245535 1 gather.go:177] gatherer "clusterconfig" function "image" took 33.764167ms to process 1 records I0405 15:04:46.245548 1 tasks_processing.go:74] worker 18 stopped. I0405 15:04:46.245717 1 tasks_processing.go:74] worker 28 stopped. I0405 15:04:46.245817 1 recorder.go:75] Recording config/featuregate with fingerprint=316cd4c430bf64b21824777d582251b2d8c66950a20892fc2f20867b6099cf9a I0405 15:04:46.245837 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 33.908344ms to process 1 records I0405 15:04:46.245930 1 tasks_processing.go:74] worker 51 stopped. I0405 15:04:46.246037 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=1dd10f42e27e1bbdd6dc7339048e1a5a83e3b3dd5e5a0e0ae9d168e72b459962 I0405 15:04:46.246438 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 34.158459ms to process 1 records I0405 15:04:46.246527 1 tasks_processing.go:74] worker 58 stopped. I0405 15:04:46.246627 1 recorder.go:75] Recording config/authentication with fingerprint=b68b98075e4fb2780e875bfceb6203e3e5f549f47c87e9529a120bcd71d6c201 I0405 15:04:46.246644 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 34.418323ms to process 1 records I0405 15:04:46.246658 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 38.030634ms to process 0 records I0405 15:04:46.246667 1 tasks_processing.go:74] worker 12 stopped. W0405 15:04:46.249470 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 15:04:46.251744 1 tasks_processing.go:74] worker 52 stopped. I0405 15:04:46.251831 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=e9822bef96acc3c5339350767b1cda0adc340cfe2f152274315c671ae9a0706f I0405 15:04:46.251851 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 40.771763ms to process 1 records I0405 15:04:46.251927 1 tasks_processing.go:74] worker 15 stopped. I0405 15:04:46.252416 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=a823030c90eafe1cf09f8ee1c3b2d41ccaa23e8cecb51ba08ed5765a869393a7 I0405 15:04:46.252609 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=662437693ea823e57401514adc6d4f73dcc2c4210bc71e9f25180f829b3021f6 I0405 15:04:46.252619 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.345859ms to process 2 records I0405 15:04:46.252633 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 41.625483ms to process 0 records I0405 15:04:46.252645 1 tasks_processing.go:74] worker 5 stopped. I0405 15:04:46.252714 1 tasks_processing.go:74] worker 49 stopped. I0405 15:04:46.252732 1 recorder.go:75] Recording config/network with fingerprint=20d0bbe47509812ff33a3168fe98ca1d575008679a623b48bbc06175a71352fe I0405 15:04:46.252737 1 gather.go:177] gatherer "clusterconfig" function "networks" took 41.415783ms to process 1 records I0405 15:04:46.252817 1 tasks_processing.go:74] worker 46 stopped. I0405 15:04:46.252963 1 recorder.go:75] Recording config/node/ip-10-0-0-121.ec2.internal with fingerprint=c167ebbfe2b2c66e5cb77857ba081961691be5e9028460c8ed338ccb3f997971 I0405 15:04:46.253006 1 recorder.go:75] Recording config/node/ip-10-0-2-78.ec2.internal with fingerprint=6f4c9270a80daa1c1478112ec2f27fe488e2872bf011e85524193efb5a5fe7bf I0405 15:04:46.253014 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 43.491989ms to process 2 records I0405 15:04:46.253093 1 tasks_processing.go:74] worker 33 stopped. I0405 15:04:46.253151 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=bbc4c52e5afa7c738b37811cc00689c93143fe572d90e14f8466f3ad9bdc4a6d I0405 15:04:46.253188 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=056e7bb50ea530a7a98f9af2de9e23fd58aaba72b158c6e86817fdd4cdf3beee I0405 15:04:46.253213 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=25be4ed822e1e89e1d73a18e5e96d16fd6f046c087f067a6f3a63495042cbb8c I0405 15:04:46.253219 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 41.811919ms to process 3 records I0405 15:04:46.253272 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=624f4dd5dbc302fb101eadf899477740b19f69a87d5c8f5720f7b914572ade89 I0405 15:04:46.253285 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=010827301e33c23131b05dfb2fa4ce70cb9d3747f2e5bf1987d24180259a7c70 I0405 15:04:46.253290 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 41.402512ms to process 2 records I0405 15:04:46.253300 1 tasks_processing.go:74] worker 53 stopped. I0405 15:04:46.254929 1 tasks_processing.go:74] worker 7 stopped. I0405 15:04:46.254959 1 gather_logs.go:145] no pods in namespace were found I0405 15:04:46.255155 1 recorder.go:75] Recording config/version with fingerprint=537a1436c40a6ce2c77631a6e361b568a6cfa4705481e8ba7635269faccbf0bb I0405 15:04:46.255170 1 recorder.go:75] Recording config/id with fingerprint=ce3735bef51404cd93ff960b44486f978f3e794cc75a977151c46643eac34dc1 I0405 15:04:46.255176 1 gather.go:177] gatherer "clusterconfig" function "version" took 45.303769ms to process 2 records I0405 15:04:46.255190 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 44.072412ms to process 0 records I0405 15:04:46.255201 1 tasks_processing.go:74] worker 20 stopped. I0405 15:04:46.256632 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0405 15:04:46.256637 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0405 15:04:46.256601 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0405 15:04:46.260783 1 base_controller.go:82] Caches are synced for ConfigController I0405 15:04:46.260797 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0405 15:04:46.262169 1 tasks_processing.go:74] worker 11 stopped. I0405 15:04:46.262242 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0405 15:04:46.262255 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0405 15:04:46.262410 1 operator.go:288] started I0405 15:04:46.262411 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=5bd338fc4695d1eb4b0a62956135a80a8093ba38ec79f14929798a2b031d385b I0405 15:04:46.262429 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0405 15:04:46.262570 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2072cbda5e051626c7c2f0c2a4923c6eb717a23a975121b1ac98706259187310 I0405 15:04:46.262593 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 53.451912ms to process 2 records I0405 15:04:46.262674 1 tasks_processing.go:74] worker 17 stopped. I0405 15:04:46.262712 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=cd3fb2380b4c1e9c3f9aafab65d32d139e2ee552faaea70a1845c80790b44229 I0405 15:04:46.262783 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=63db237fcaf7cb7fd7da6ff3bfc67b182f307777a60a7483e6a8a790196d6e5a I0405 15:04:46.262802 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=ad16ebe20b66813857cb0cc6ac1869baa728d55cb69c1923b1d163e3fdf3e0b3 I0405 15:04:46.262826 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=b9d18f7c5a2d4f2f8946dff11ad9145469e065f5cbb50999fb386227b5e5cf61 I0405 15:04:46.262849 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=90df2a93ed936d6decde54c0549f58f12c319fe2c2614f62906b5d624fb4c372 I0405 15:04:46.262871 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=2340ca69e465610ff084f97dcfd2eefc0e7bc723a20ae98f29a07d9a3b21e043 I0405 15:04:46.262894 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=3ef37e601767fbb6313230f6d489aa89292caf16f456aaa20843a703620aa179 I0405 15:04:46.262926 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=799c8b62031b388d4c6a0275d1b0a6a20872e328b6a391d235fc8f7496a2d622 I0405 15:04:46.262948 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=1ad8e2738dfeece464c0c747492d55b7bb566b6c10c15a2f854d58d1f6390add I0405 15:04:46.262971 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=897c2d9e5dd12d6d25a1f64cacac5bc4519cee1d5b68be9f7a40fe42f8fd32fe I0405 15:04:46.262991 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=c174683b39d13ba7f751dfafffcd26efc784920027dbf3e7a99cc77b00cf3175 I0405 15:04:46.262997 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 51.333701ms to process 11 records I0405 15:04:46.266770 1 tasks_processing.go:74] worker 44 stopped. I0405 15:04:46.266783 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 56.234115ms to process 0 records I0405 15:04:46.268682 1 tasks_processing.go:74] worker 45 stopped. I0405 15:04:46.268768 1 recorder.go:75] Recording config/running_containers with fingerprint=4fc273040d0b82334f1fcc5d2cb02595c21e0656f97f590a0d3aa8a89cd69bc7 I0405 15:04:46.268786 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 57.629511ms to process 1 records I0405 15:04:46.270342 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 15:04:46.279680 1 tasks_processing.go:74] worker 26 stopped. I0405 15:04:46.279700 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 69.968861ms to process 0 records I0405 15:04:46.299765 1 prometheus_rules.go:88] Prometheus rules successfully created I0405 15:04:46.300349 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0405 15:04:46.300360 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0405 15:04:46.300363 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0405 15:04:46.300366 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0405 15:04:46.300369 1 controller.go:212] Source scaController *sca.Controller is not ready I0405 15:04:46.300384 1 controller.go:489] The operator is still being initialized I0405 15:04:46.300392 1 controller.go:512] The operator is healthy E0405 15:04:46.304062 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%270f5da126-cd2c-4c50-accb-0c226e203baa%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:38506->172.30.0.10:53: read: connection refused I0405 15:04:46.304076 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%270f5da126-cd2c-4c50-accb-0c226e203baa%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:38506->172.30.0.10:53: read: connection refused I0405 15:04:46.318028 1 tasks_processing.go:74] worker 22 stopped. E0405 15:04:46.318040 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0405 15:04:46.318046 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0405 15:04:46.318050 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0405 15:04:46.318059 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0405 15:04:46.318081 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0405 15:04:46.318085 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0405 15:04:46.318089 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0405 15:04:46.318093 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0405 15:04:46.318143 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0405 15:04:46.318151 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0405 15:04:46.318156 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 108.103223ms to process 7 records I0405 15:04:46.329519 1 tasks_processing.go:74] worker 60 stopped. I0405 15:04:46.329539 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 118.508254ms to process 0 records I0405 15:04:46.348634 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0405 15:04:46.351733 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:46490->172.30.0.10:53: read: connection refused I0405 15:04:46.351745 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:46490->172.30.0.10:53: read: connection refused I0405 15:04:46.362516 1 base_controller.go:82] Caches are synced for LoggingSyncer I0405 15:04:46.362527 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0405 15:04:46.367049 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0405 15:04:46.372025 1 tasks_processing.go:74] worker 36 stopped. I0405 15:04:46.372031 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found E0405 15:04:46.372039 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0405 15:04:46.372044 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pg2fk914niovfrjknfofsnhjcuj998d-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pg2fk914niovfrjknfofsnhjcuj998d-primary-cert-bundle-secret" not found I0405 15:04:46.372098 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=f256a6095f06eed0b101392a594b1188ec5f7b3aff36ee3cd10bdbf7b1b7557e I0405 15:04:46.372112 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 162.988626ms to process 1 records I0405 15:04:46.671895 1 gather_cluster_operator_pods_and_events.go:121] Found 11 pods with 12 containers I0405 15:04:46.671907 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2097152 bytes I0405 15:04:46.672137 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-2q6zt pod in namespace openshift-dns (previous: false). I0405 15:04:46.906382 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-2q6zt pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-2q6zt\" is waiting to start: ContainerCreating" I0405 15:04:46.906401 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-2q6zt\" is waiting to start: ContainerCreating" I0405 15:04:46.906413 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-2q6zt pod in namespace openshift-dns (previous: false). I0405 15:04:47.079788 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-2q6zt pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-2q6zt\" is waiting to start: ContainerCreating" I0405 15:04:47.079806 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-2q6zt\" is waiting to start: ContainerCreating" I0405 15:04:47.079815 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-g69tk pod in namespace openshift-dns (previous: false). W0405 15:04:47.248330 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 15:04:47.265895 1 tasks_processing.go:74] worker 29 stopped. I0405 15:04:47.265915 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 1.057079062s to process 0 records I0405 15:04:47.272435 1 tasks_processing.go:74] worker 32 stopped. I0405 15:04:47.272469 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0405 15:04:47.272481 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 1.061924259s to process 1 records I0405 15:04:47.277671 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 15:04:47.277685 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-xcqdw pod in namespace openshift-dns (previous: false). I0405 15:04:47.504059 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-xcqdw pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "the server could not find the requested resource ( pods/log node-resolver-xcqdw)" I0405 15:04:47.504075 1 gather_cluster_operator_pods_and_events.go:280] Error: "the server could not find the requested resource ( pods/log node-resolver-xcqdw)" I0405 15:04:47.504087 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6cf4c885df-9gqq2 pod in namespace openshift-image-registry (previous: false). I0405 15:04:47.672345 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 15:04:47.672368 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6f84f459d6-6nl4f pod in namespace openshift-image-registry (previous: false). I0405 15:04:47.690830 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0405 15:04:47.879660 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6f84f459d6-6nl4f pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6f84f459d6-6nl4f\" is waiting to start: ContainerCreating" I0405 15:04:47.879680 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6f84f459d6-6nl4f\" is waiting to start: ContainerCreating" I0405 15:04:47.879689 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6f84f459d6-zxjcr pod in namespace openshift-image-registry (previous: false). I0405 15:04:48.073441 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 15:04:48.073467 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-nl46s pod in namespace openshift-image-registry (previous: false). I0405 15:04:48.093806 1 tasks_processing.go:74] worker 13 stopped. I0405 15:04:48.093855 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=7b54240ed581a0a55d244c6005e00a0a6c2e7cf663e16dae64fc09d625a0ca68 I0405 15:04:48.093887 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=59fccadacf38589c6b40ea803f41a59ad1a8fe42fe73f5511835096c7e826cc3 I0405 15:04:48.093922 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0405 15:04:48.093951 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=6781d8f783c0e7fa4d632e1b57bf65ba5d87d407afb6924f97c0b8315aeeaf06 I0405 15:04:48.093967 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0405 15:04:48.093997 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=59b9c1393d71f21cfd6cd43df44902eb5d5f1a991a35ae828ea78b3bcbca93bb I0405 15:04:48.094054 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=5e077e2fcf80fca95c5e5deb9a277bc2c72304c367703925d2176c829195305a I0405 15:04:48.094109 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=40d1122f08dd3f58d8773f0daa4571d482dc6de783b686d1c6d48056bf31a14d I0405 15:04:48.094145 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=4f0f75da68b98f56682dee40ff72f6057be67edbe0e055135844aa77c6d0c9cd I0405 15:04:48.094175 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=9e113b33d92371f90d85510ca049f33509661875a0695b6cc1dbde6cafa7ca13 I0405 15:04:48.094188 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0405 15:04:48.094205 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=cfadd39103ff92521bc483071113270181d2c83760f614d2af5be42be37a14c7 I0405 15:04:48.094219 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0405 15:04:48.094246 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=de13db46b70408355e99cfe5c205063bbd104314847708737aa0149ee5b63a03 I0405 15:04:48.094259 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0405 15:04:48.094279 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=bb907750ad62b655570c36930c73818f653f26004e7533fe7f345b2966d6f319 I0405 15:04:48.094294 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0405 15:04:48.094318 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=4642fac577e7b9efda922a5cedc1ae930c4b5ec2e709f75e5c7c8b75d78afc67 I0405 15:04:48.094520 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=45fa314b4305857e41c14c8278016df728204382727174057a5f12160bbd07a4 I0405 15:04:48.094534 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0405 15:04:48.094545 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0405 15:04:48.094577 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0405 15:04:48.094612 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=89bd5901f0bab510a2544ab0c7f6b17e6ef49c2a5617589a215e762000d9086e I0405 15:04:48.094653 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=48d4e811b60291063112e501d3b94823732c818005aabc8a4482b977416d19d2 I0405 15:04:48.094667 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0405 15:04:48.094692 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=cccc6fd21132836a4cec84e7d457a6c43c9e33f79fc97ed826264bb9afecfcda I0405 15:04:48.094707 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0405 15:04:48.094730 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=dc15f3b1e712fa70b0e18087cc5195ae784995a378798da45f2d4d876c424a5a I0405 15:04:48.094752 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=46921d0653ef66e8abfe18b646a5bb35235a9942052b9764d87dae055a3d4d90 I0405 15:04:48.094778 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=f5a60d036fa4c9ad7325f1655ccc3c680d6848a11b42b5f7b944911e3842e7aa I0405 15:04:48.094803 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=14d43815a5e4cab21977594d2aae447e6b1f6e32379f3a7c8f851198f12fed26 I0405 15:04:48.094822 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=5cb638d30e597c088defb731db831cdb493b9f58d86ba6138be1a81c4eed8a39 I0405 15:04:48.094863 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=d24554881b268d793c3052e16a3edc0ca563df4dae95cbbf2bbc1c9076dbe24d I0405 15:04:48.094887 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0405 15:04:48.094900 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0405 15:04:48.094910 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.88450201s to process 35 records W0405 15:04:48.248977 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0405 15:04:48.277430 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 15:04:48.277446 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-596cbdf8f5-vg5j2 pod in namespace openshift-ingress (previous: false). I0405 15:04:48.491357 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-596cbdf8f5-vg5j2 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-596cbdf8f5-vg5j2\" is waiting to start: ContainerCreating" I0405 15:04:48.491378 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-596cbdf8f5-vg5j2\" is waiting to start: ContainerCreating" I0405 15:04:48.491388 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7f88857fbd-7sls6 pod in namespace openshift-ingress (previous: false). I0405 15:04:48.682377 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7f88857fbd-7sls6 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7f88857fbd-7sls6\" is waiting to start: ContainerCreating" I0405 15:04:48.682393 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7f88857fbd-7sls6\" is waiting to start: ContainerCreating" I0405 15:04:48.682404 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7f88857fbd-dh6bv pod in namespace openshift-ingress (previous: false). I0405 15:04:48.884679 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0405 15:04:48.884695 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-47lbg pod in namespace openshift-ingress-canary (previous: false). I0405 15:04:49.077917 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-47lbg pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-47lbg\" is waiting to start: ContainerCreating" I0405 15:04:49.077933 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-47lbg\" is waiting to start: ContainerCreating" I0405 15:04:49.077948 1 tasks_processing.go:74] worker 0 stopped. I0405 15:04:49.078022 1 recorder.go:75] Recording events/openshift-dns with fingerprint=f9b261686fdd75c74372fd636700c85d5e7a0fb41012a106d093b5a3847b9c5d I0405 15:04:49.078095 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=62a5ddaa49f793594ca7ae2249b17f26b9ee28fee72601d706dd10ce05b1ee05 I0405 15:04:49.078134 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=2eedb88e33bcf53bf3fdb9d947f8dc1c8095dba64410997d3a73c30909adade1 I0405 15:04:49.078177 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=32768b139070c933cda51ea5011fe706a1e7d62965b0e32d20f022ee8c536ca2 I0405 15:04:49.078189 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=f4e8ca9b8b632dce33128e0ba9aaa436e073498de8efe51969684416ce016cee I0405 15:04:49.078197 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.868168874s to process 5 records W0405 15:04:49.248608 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0405 15:04:50.248745 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0405 15:04:51.244951 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0405 15:04:51.244976 1 tasks_processing.go:74] worker 2 stopped. E0405 15:04:51.244986 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0405 15:04:51.245022 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0405 15:04:51.245039 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0405 15:04:51.245050 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.036375107s to process 1 records I0405 15:04:58.651276 1 tasks_processing.go:74] worker 8 stopped. I0405 15:04:58.651313 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0405 15:04:58.651324 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.442705937s to process 1 records I0405 15:04:59.375583 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 15:04:59.413872 1 tasks_processing.go:74] worker 19 stopped. I0405 15:04:59.414150 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=5f3315562ab922847b80910efb7b38b1b30885aa3cdda0895bd6e02e66f43fde I0405 15:04:59.414166 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.204773542s to process 1 records E0405 15:04:59.414223 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.205s with: function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0405 15:04:59.415331 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0405 15:04:59.415346 1 periodic.go:209] Running workloads gatherer I0405 15:04:59.415364 1 tasks_processing.go:45] number of workers: 2 I0405 15:04:59.415377 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 15:04:59.415385 1 tasks_processing.go:71] worker 1 working on workload_info task. I0405 15:04:59.415391 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 15:04:59.415413 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0405 15:04:59.436092 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 20s for image data I0405 15:04:59.437656 1 tasks_processing.go:74] worker 0 stopped. I0405 15:04:59.437671 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 22.22151ms to process 0 records I0405 15:04:59.443221 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (8ms) I0405 15:04:59.450275 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (7ms) I0405 15:04:59.457232 1 gather_workloads_info.go:387] No image sha256:eff3318d5ac1d3a141221b536233305f54b3334fab56e5f8ba20ef8a9ed21429 (7ms) I0405 15:04:59.464869 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0405 15:04:59.472621 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (8ms) I0405 15:04:59.480180 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (8ms) I0405 15:04:59.486953 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (7ms) I0405 15:04:59.494471 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (8ms) I0405 15:04:59.501611 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (7ms) I0405 15:04:59.508477 1 gather_workloads_info.go:387] No image sha256:d3209a996e5ce362136e70a0bcf06d89973905de73e234b644d8ae8886d067f1 (7ms) I0405 15:04:59.545731 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (37ms) I0405 15:04:59.643922 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (98ms) I0405 15:04:59.744149 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0405 15:04:59.843443 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (99ms) I0405 15:04:59.944364 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (101ms) I0405 15:05:00.043151 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (99ms) I0405 15:05:00.143152 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0405 15:05:00.244138 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (101ms) I0405 15:05:00.344974 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (101ms) I0405 15:05:00.444921 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0405 15:05:00.545457 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (101ms) I0405 15:05:00.644971 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (99ms) I0405 15:05:00.746078 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0405 15:05:00.845457 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (99ms) I0405 15:05:00.946199 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0405 15:05:01.044901 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (99ms) I0405 15:05:01.144824 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0405 15:05:01.247708 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (103ms) I0405 15:05:01.346397 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (99ms) I0405 15:05:01.444147 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (98ms) I0405 15:05:01.543395 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0405 15:05:01.643435 1 gather_workloads_info.go:387] No image sha256:da4f8a01ef5d0607cf76fa7b219fafdb6020cb4cf9a0a88cfe76d292c1a561fc (100ms) I0405 15:05:01.744287 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (101ms) I0405 15:05:01.843917 1 gather_workloads_info.go:387] No image sha256:ef2bfb546cd8aac31b7671585f718f44b268ed43a9c8562016ab741558a7dfa1 (100ms) I0405 15:05:01.944514 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (101ms) I0405 15:05:02.044929 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0405 15:05:02.144476 1 gather_workloads_info.go:387] No image sha256:a505dd5b62d744c2cfda2666b5e21e0a4704354a3114488bd3291456085e1e33 (100ms) I0405 15:05:02.243561 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (99ms) I0405 15:05:02.343834 1 gather_workloads_info.go:387] No image sha256:fab570ecdc9c3620b032bc4c5b44240f5ebac0bb9e48dfeea32bcd36eb462e5a (100ms) I0405 15:05:02.343861 1 tasks_processing.go:74] worker 1 stopped. E0405 15:05:02.343870 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0405 15:05:02.344134 1 recorder.go:75] Recording config/workload_info with fingerprint=7fd4f8ec770b6d400779b3fa1884f7acbe1de0fbbcb0768ebd8a8bb9a4908b14 I0405 15:05:02.344148 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.92846848s to process 1 records E0405 15:05:02.344171 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.928s with: function \"workload_info\" failed with an error" I0405 15:05:02.345266 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0405 15:05:02.345278 1 periodic.go:209] Running conditional gatherer I0405 15:05:02.349647 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0405 15:05:02.355819 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.19:34773->172.30.0.10:53: read: connection refused E0405 15:05:02.356054 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0405 15:05:02.356106 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0405 15:05:02.360630 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0405 15:05:02.360643 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360648 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360651 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360654 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360657 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360660 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360663 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360666 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0405 15:05:02.360669 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0405 15:05:02.360681 1 tasks_processing.go:45] number of workers: 3 I0405 15:05:02.360691 1 tasks_processing.go:69] worker 2 listening for tasks. I0405 15:05:02.360695 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0405 15:05:02.360700 1 tasks_processing.go:69] worker 0 listening for tasks. I0405 15:05:02.360708 1 tasks_processing.go:69] worker 1 listening for tasks. I0405 15:05:02.360730 1 tasks_processing.go:71] worker 1 working on conditional_gatherer_rules task. I0405 15:05:02.360719 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0405 15:05:02.360754 1 tasks_processing.go:74] worker 1 stopped. I0405 15:05:02.360825 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0405 15:05:02.360837 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.288µs to process 1 records I0405 15:05:02.360871 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0405 15:05:02.360878 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.529µs to process 1 records I0405 15:05:02.360884 1 tasks_processing.go:74] worker 0 stopped. I0405 15:05:02.360940 1 tasks_processing.go:74] worker 2 stopped. I0405 15:05:02.360952 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 240.397µs to process 0 records I0405 15:05:02.360991 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.19:34773->172.30.0.10:53: read: connection refused I0405 15:05:02.361010 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0405 15:05:02.381356 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=2235f46644d4800d67904bc55b0b4e4b02162e30bdad50f952b607741394a861 I0405 15:05:02.381494 1 diskrecorder.go:70] Writing 97 records to /var/lib/insights-operator/insights-2026-04-05-150502.tar.gz I0405 15:05:02.387455 1 diskrecorder.go:51] Wrote 97 records to disk in 6ms I0405 15:05:02.387484 1 periodic.go:278] Gathering cluster info every 2h0m0s I0405 15:05:02.387499 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0405 15:05:02.437175 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 15:05:02.636574 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 15:05:12.539873 1 configmapobserver.go:84] configmaps "insights-config" not found I0405 15:05:55.786295 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="9431b414c1c1dbb7a6d4302a49c1e8c2ba5a64241a077b6e09bc890024583516") W0405 15:05:55.786326 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0405 15:05:55.786370 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="ed5818dd31838cf3996b03d3edcc2c2e1f9c3cbe716baaccad7d4e99b61ded24") I0405 15:05:55.786399 1 base_controller.go:181] Shutting down LoggingSyncer ... I0405 15:05:55.786395 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0405 15:05:55.786430 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0405 15:05:55.786443 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0405 15:05:55.786448 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="d50253bbae229793a56d1c1704941d616f84ed6e141b63499e84237c64bc9268") I0405 15:05:55.786452 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0405 15:05:55.786434 1 periodic.go:170] Shutting down I0405 15:05:55.786465 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0405 15:05:55.786490 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0405 15:05:55.786488 1 base_controller.go:181] Shutting down ConfigController ... I0405 15:05:55.786502 1 base_controller.go:113] All ConfigController workers have been terminated I0405 15:05:55.786514 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting"