W0330 14:59:46.751027 1 cmd.go:257] Using insecure, self-signed certificates I0330 14:59:46.964802 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 14:59:46.965112 1 observer_polling.go:159] Starting file observer I0330 14:59:47.725443 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0330 14:59:47.725735 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0330 14:59:47.726596 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0330 14:59:47.726677 1 secure_serving.go:57] Forcing use of http/1.1 only W0330 14:59:47.726705 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0330 14:59:47.726716 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0330 14:59:47.726726 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0330 14:59:47.726730 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0330 14:59:47.726734 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0330 14:59:47.726738 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0330 14:59:47.731565 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0330 14:59:47.732117 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"573319cb-806e-4a27-91c2-172c8dfb0cab", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0330 14:59:47.733186 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0330 14:59:47.733208 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0330 14:59:47.733204 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0330 14:59:47.733209 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0330 14:59:47.733224 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0330 14:59:47.733227 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0330 14:59:47.733542 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1917226416/tls.crt::/tmp/serving-cert-1917226416/tls.key" I0330 14:59:47.733969 1 secure_serving.go:213] Serving securely on [::]:8443 I0330 14:59:47.734021 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0330 14:59:47.737056 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0330 14:59:47.737081 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0330 14:59:47.737217 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0330 14:59:47.743785 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0330 14:59:47.743805 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0330 14:59:47.749043 1 secretconfigobserver.go:119] support secret does not exist I0330 14:59:47.753492 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0330 14:59:47.759421 1 secretconfigobserver.go:119] support secret does not exist I0330 14:59:47.763121 1 recorder.go:176] Pruning old reports every 6h14m18s, max age is 288h0m0s I0330 14:59:47.772002 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0330 14:59:47.772023 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0330 14:59:47.772031 1 periodic.go:216] Running clusterconfig gatherer I0330 14:59:47.772034 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0330 14:59:47.772080 1 tasks_processing.go:45] number of workers: 64 I0330 14:59:47.772116 1 tasks_processing.go:69] worker 8 listening for tasks. I0330 14:59:47.772034 1 insightsreport.go:296] Starting report retriever I0330 14:59:47.772132 1 tasks_processing.go:69] worker 21 listening for tasks. I0330 14:59:47.772138 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0330 14:59:47.772143 1 tasks_processing.go:71] worker 21 working on machine_config_pools task. I0330 14:59:47.772145 1 tasks_processing.go:69] worker 34 listening for tasks. I0330 14:59:47.772154 1 tasks_processing.go:69] worker 0 listening for tasks. I0330 14:59:47.772155 1 tasks_processing.go:69] worker 15 listening for tasks. I0330 14:59:47.772162 1 tasks_processing.go:69] worker 17 listening for tasks. I0330 14:59:47.772163 1 tasks_processing.go:71] worker 0 working on monitoring_persistent_volumes task. I0330 14:59:47.772170 1 tasks_processing.go:71] worker 17 working on storage_classes task. I0330 14:59:47.772173 1 tasks_processing.go:69] worker 2 listening for tasks. I0330 14:59:47.772174 1 tasks_processing.go:69] worker 10 listening for tasks. I0330 14:59:47.772164 1 tasks_processing.go:71] worker 15 working on certificate_signing_requests task. I0330 14:59:47.772182 1 tasks_processing.go:69] worker 11 listening for tasks. I0330 14:59:47.772191 1 tasks_processing.go:69] worker 12 listening for tasks. I0330 14:59:47.772186 1 tasks_processing.go:69] worker 49 listening for tasks. I0330 14:59:47.772190 1 tasks_processing.go:69] worker 28 listening for tasks. I0330 14:59:47.772199 1 tasks_processing.go:69] worker 13 listening for tasks. I0330 14:59:47.772207 1 tasks_processing.go:69] worker 35 listening for tasks. I0330 14:59:47.772156 1 tasks_processing.go:71] worker 34 working on ingress task. I0330 14:59:47.772217 1 tasks_processing.go:69] worker 22 listening for tasks. I0330 14:59:47.772218 1 tasks_processing.go:69] worker 36 listening for tasks. I0330 14:59:47.772217 1 tasks_processing.go:69] worker 31 listening for tasks. I0330 14:59:47.772226 1 tasks_processing.go:69] worker 37 listening for tasks. I0330 14:59:47.772231 1 tasks_processing.go:69] worker 23 listening for tasks. I0330 14:59:47.772231 1 tasks_processing.go:69] worker 29 listening for tasks. I0330 14:59:47.772235 1 tasks_processing.go:69] worker 38 listening for tasks. I0330 14:59:47.772239 1 tasks_processing.go:69] worker 30 listening for tasks. I0330 14:59:47.772246 1 tasks_processing.go:69] worker 26 listening for tasks. I0330 14:59:47.772241 1 tasks_processing.go:69] worker 39 listening for tasks. I0330 14:59:47.772249 1 tasks_processing.go:69] worker 40 listening for tasks. I0330 14:59:47.772255 1 tasks_processing.go:69] worker 53 listening for tasks. I0330 14:59:47.772259 1 tasks_processing.go:69] worker 24 listening for tasks. I0330 14:59:47.772258 1 tasks_processing.go:69] worker 57 listening for tasks. I0330 14:59:47.772265 1 tasks_processing.go:69] worker 50 listening for tasks. I0330 14:59:47.772260 1 tasks_processing.go:69] worker 41 listening for tasks. I0330 14:59:47.772267 1 tasks_processing.go:69] worker 42 listening for tasks. I0330 14:59:47.772272 1 tasks_processing.go:69] worker 33 listening for tasks. I0330 14:59:47.772274 1 tasks_processing.go:69] worker 54 listening for tasks. I0330 14:59:47.772268 1 tasks_processing.go:69] worker 27 listening for tasks. I0330 14:59:47.772274 1 tasks_processing.go:69] worker 46 listening for tasks. I0330 14:59:47.772284 1 tasks_processing.go:69] worker 55 listening for tasks. I0330 14:59:47.772284 1 tasks_processing.go:69] worker 52 listening for tasks. I0330 14:59:47.772288 1 tasks_processing.go:69] worker 48 listening for tasks. I0330 14:59:47.772282 1 tasks_processing.go:69] worker 47 listening for tasks. I0330 14:59:47.772293 1 tasks_processing.go:69] worker 56 listening for tasks. I0330 14:59:47.772150 1 tasks_processing.go:69] worker 16 listening for tasks. I0330 14:59:47.772282 1 tasks_processing.go:69] worker 32 listening for tasks. I0330 14:59:47.772277 1 tasks_processing.go:69] worker 51 listening for tasks. I0330 14:59:47.772302 1 tasks_processing.go:69] worker 61 listening for tasks. I0330 14:59:47.772303 1 tasks_processing.go:69] worker 44 listening for tasks. I0330 14:59:47.772287 1 tasks_processing.go:69] worker 45 listening for tasks. I0330 14:59:47.772309 1 tasks_processing.go:69] worker 3 listening for tasks. I0330 14:59:47.772311 1 tasks_processing.go:69] worker 43 listening for tasks. I0330 14:59:47.772312 1 tasks_processing.go:69] worker 62 listening for tasks. I0330 14:59:47.772317 1 tasks_processing.go:69] worker 6 listening for tasks. I0330 14:59:47.772319 1 tasks_processing.go:69] worker 4 listening for tasks. I0330 14:59:47.772317 1 tasks_processing.go:69] worker 60 listening for tasks. I0330 14:59:47.772322 1 tasks_processing.go:71] worker 8 working on infrastructures task. I0330 14:59:47.772358 1 tasks_processing.go:69] worker 25 listening for tasks. I0330 14:59:47.772142 1 tasks_processing.go:69] worker 9 listening for tasks. I0330 14:59:47.772163 1 tasks_processing.go:69] worker 1 listening for tasks. I0330 14:59:47.772301 1 tasks_processing.go:69] worker 5 listening for tasks. I0330 14:59:47.772303 1 tasks_processing.go:69] worker 63 listening for tasks. I0330 14:59:47.772293 1 tasks_processing.go:69] worker 18 listening for tasks. I0330 14:59:47.772310 1 tasks_processing.go:69] worker 58 listening for tasks. I0330 14:59:47.772313 1 tasks_processing.go:69] worker 59 listening for tasks. I0330 14:59:47.772322 1 tasks_processing.go:69] worker 7 listening for tasks. I0330 14:59:47.772336 1 tasks_processing.go:69] worker 20 listening for tasks. I0330 14:59:47.772208 1 tasks_processing.go:69] worker 14 listening for tasks. I0330 14:59:47.772415 1 tasks_processing.go:71] worker 37 working on clusterroles task. I0330 14:59:47.772422 1 tasks_processing.go:71] worker 36 working on machines task. I0330 14:59:47.772425 1 tasks_processing.go:71] worker 49 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0330 14:59:47.772430 1 tasks_processing.go:71] worker 11 working on container_images task. I0330 14:59:47.772431 1 tasks_processing.go:71] worker 22 working on install_plans task. I0330 14:59:47.772437 1 tasks_processing.go:71] worker 60 working on pdbs task. I0330 14:59:47.772445 1 tasks_processing.go:71] worker 9 working on openstack_controlplanes task. I0330 14:59:47.772453 1 tasks_processing.go:71] worker 38 working on pod_network_connectivity_checks task. I0330 14:59:47.772460 1 tasks_processing.go:71] worker 31 working on metrics task. W0330 14:59:47.772497 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 14:59:47.772511 1 tasks_processing.go:71] worker 1 working on schedulers task. I0330 14:59:47.772514 1 tasks_processing.go:71] worker 30 working on lokistack task. I0330 14:59:47.772505 1 tasks_processing.go:71] worker 18 working on node_logs task. I0330 14:59:47.772427 1 tasks_processing.go:71] worker 23 working on mutating_webhook_configurations task. I0330 14:59:47.772655 1 tasks_processing.go:71] worker 20 working on openstack_version task. I0330 14:59:47.772435 1 tasks_processing.go:71] worker 32 working on sap_config task. I0330 14:59:47.772415 1 tasks_processing.go:71] worker 35 working on olm_operators task. I0330 14:59:47.772441 1 tasks_processing.go:71] worker 25 working on version task. I0330 14:59:47.772431 1 tasks_processing.go:71] worker 29 working on qemu_kubevirt_launcher_logs task. I0330 14:59:47.772656 1 tasks_processing.go:71] worker 33 working on overlapping_namespace_uids task. I0330 14:59:47.772446 1 tasks_processing.go:71] worker 16 working on container_runtime_configs task. I0330 14:59:47.772291 1 tasks_processing.go:69] worker 19 listening for tasks. I0330 14:59:47.772430 1 tasks_processing.go:71] worker 10 working on machine_autoscalers task. I0330 14:59:47.772517 1 tasks_processing.go:71] worker 31 working on nodenetworkstates task. I0330 14:59:47.772423 1 tasks_processing.go:71] worker 2 working on authentication task. I0330 14:59:47.772421 1 tasks_processing.go:71] worker 13 working on operators task. I0330 14:59:47.772620 1 tasks_processing.go:71] worker 42 working on image_pruners task. I0330 14:59:47.772609 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 43.561µs to process 0 records I0330 14:59:47.772636 1 tasks_processing.go:71] worker 40 working on tsdb_status task. I0330 14:59:47.772637 1 tasks_processing.go:71] worker 59 working on openshift_machine_api_events task. I0330 14:59:47.772640 1 tasks_processing.go:71] worker 53 working on oauths task. I0330 14:59:47.772644 1 tasks_processing.go:71] worker 24 working on sap_datahubs task. I0330 14:59:47.772643 1 tasks_processing.go:71] worker 58 working on config_maps task. I0330 14:59:47.772648 1 tasks_processing.go:71] worker 57 working on sap_pods task. I0330 14:59:47.772649 1 tasks_processing.go:71] worker 7 working on cluster_apiserver task. I0330 14:59:47.772652 1 tasks_processing.go:71] worker 50 working on machine_healthchecks task. I0330 14:59:47.772652 1 tasks_processing.go:71] worker 55 working on networks task. I0330 14:59:47.772656 1 tasks_processing.go:71] worker 41 working on openstack_dataplanedeployments task. I0330 14:59:47.772660 1 tasks_processing.go:71] worker 54 working on support_secret task. I0330 14:59:47.772661 1 tasks_processing.go:71] worker 51 working on dvo_metrics task. I0330 14:59:47.772664 1 tasks_processing.go:71] worker 27 working on image_registries task. I0330 14:59:47.772664 1 tasks_processing.go:71] worker 43 working on ceph_cluster task. I0330 14:59:47.772666 1 tasks_processing.go:71] worker 62 working on validating_webhook_configurations task. I0330 14:59:47.772668 1 tasks_processing.go:71] worker 61 working on image task. I0330 14:59:47.772667 1 tasks_processing.go:71] worker 46 working on feature_gates task. I0330 14:59:47.772672 1 tasks_processing.go:71] worker 6 working on cost_management_metrics_configs task. I0330 14:59:47.772669 1 tasks_processing.go:71] worker 5 working on jaegers task. I0330 14:59:47.772672 1 tasks_processing.go:71] worker 44 working on storage_cluster task. I0330 14:59:47.772677 1 tasks_processing.go:71] worker 45 working on machine_sets task. I0330 14:59:47.772674 1 tasks_processing.go:71] worker 63 working on machine_configs task. I0330 14:59:47.772677 1 tasks_processing.go:71] worker 4 working on aggregated_monitoring_cr_names task. I0330 14:59:47.772679 1 tasks_processing.go:71] worker 48 working on openstack_dataplanenodesets task. I0330 14:59:47.772681 1 tasks_processing.go:71] worker 3 working on proxies task. I0330 14:59:47.772683 1 tasks_processing.go:71] worker 47 working on nodes task. I0330 14:59:47.772683 1 tasks_processing.go:71] worker 52 working on openshift_logging task. I0330 14:59:47.772419 1 tasks_processing.go:71] worker 28 working on operators_pods_and_events task. I0330 14:59:47.772688 1 tasks_processing.go:71] worker 56 working on service_accounts task. I0330 14:59:47.772417 1 tasks_processing.go:71] worker 12 working on crds task. I0330 14:59:47.772423 1 tasks_processing.go:71] worker 14 working on silenced_alerts task. I0330 14:59:47.772968 1 tasks_processing.go:71] worker 19 working on ingress_certificates task. W0330 14:59:47.775145 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 14:59:47.775159 1 tasks_processing.go:71] worker 14 working on nodenetworkconfigurationpolicies task. I0330 14:59:47.775189 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 76.414µs to process 0 records I0330 14:59:47.772626 1 tasks_processing.go:71] worker 26 working on node_features task. I0330 14:59:47.772630 1 tasks_processing.go:71] worker 39 working on active_alerts task. W0330 14:59:47.775375 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 14:59:47.775397 1 tasks_processing.go:74] worker 39 stopped. I0330 14:59:47.775408 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 42.847µs to process 0 records W0330 14:59:47.773146 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 14:59:47.775428 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 2.307671ms to process 0 records I0330 14:59:47.775437 1 tasks_processing.go:74] worker 40 stopped. I0330 14:59:47.783112 1 tasks_processing.go:74] worker 8 stopped. I0330 14:59:47.783493 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0330 14:59:47.783517 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0330 14:59:47.783523 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0330 14:59:47.783527 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0330 14:59:47.783545 1 controller.go:531] The operator is still being initialized I0330 14:59:47.783556 1 controller.go:554] The operator is healthy I0330 14:59:47.783674 1 recorder.go:75] Recording config/infrastructure with fingerprint=72961fdcb73ef544d73b7d39235053c6d3933069894767944b8933b1f7b08082 I0330 14:59:47.783688 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 10.762126ms to process 1 records I0330 14:59:47.783698 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 10.656484ms to process 0 records I0330 14:59:47.783702 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 11.034758ms to process 0 records I0330 14:59:47.783705 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 10.039602ms to process 0 records I0330 14:59:47.783711 1 tasks_processing.go:74] worker 30 stopped. I0330 14:59:47.783720 1 tasks_processing.go:74] worker 24 stopped. I0330 14:59:47.783721 1 tasks_processing.go:74] worker 21 stopped. I0330 14:59:47.783782 1 tasks_processing.go:74] worker 34 stopped. I0330 14:59:47.783824 1 recorder.go:75] Recording config/ingress with fingerprint=4637129e54c462e661b176a2acbfaeb74d5ef10173a24be7da87f92dd16fa134 I0330 14:59:47.783832 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 11.208989ms to process 1 records I0330 14:59:47.783887 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=5ac8646173c232be9bded6f451b882902286881d85db129b5884513c0469e4b9 I0330 14:59:47.783900 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=c8081c0de64a42ee00eff48f9e740d932dcd8df1d825822cc432db93c17ac426 I0330 14:59:47.783904 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 11.279092ms to process 2 records I0330 14:59:47.783909 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 9.930802ms to process 0 records I0330 14:59:47.783911 1 tasks_processing.go:74] worker 17 stopped. I0330 14:59:47.783917 1 tasks_processing.go:74] worker 57 stopped. I0330 14:59:47.783913 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 10.187525ms to process 0 records I0330 14:59:47.783923 1 tasks_processing.go:74] worker 41 stopped. I0330 14:59:47.783942 1 tasks_processing.go:74] worker 0 stopped. I0330 14:59:47.783934 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 11.319952ms to process 0 records I0330 14:59:47.783953 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 11.1425ms to process 0 records I0330 14:59:47.783995 1 tasks_processing.go:74] worker 9 stopped. I0330 14:59:47.784062 1 tasks_processing.go:74] worker 42 stopped. I0330 14:59:47.784172 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=11aeb0809b08a1fd8dce45eed9437c6929f0850bca2ec1eb5142ccf7da7029aa I0330 14:59:47.784183 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 10.557724ms to process 1 records I0330 14:59:47.784258 1 tasks_processing.go:74] worker 53 stopped. I0330 14:59:47.784374 1 recorder.go:75] Recording config/oauth with fingerprint=d580a3140f86a9c23b00a2751bc030fd0cfc7859ff13e22417b9378329959dc7 I0330 14:59:47.784384 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 10.519478ms to process 1 records I0330 14:59:47.784459 1 tasks_processing.go:74] worker 7 stopped. I0330 14:59:47.784483 1 recorder.go:75] Recording config/apiserver with fingerprint=92874b17a5dec0e7e4624c9884fc2bbd88670cecab78271bcd50fb09a224b7d6 I0330 14:59:47.784492 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 10.421927ms to process 1 records I0330 14:59:47.785109 1 tasks_processing.go:74] worker 15 stopped. I0330 14:59:47.785125 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 12.91486ms to process 0 records I0330 14:59:47.787099 1 tasks_processing.go:74] worker 43 stopped. I0330 14:59:47.787115 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 13.417476ms to process 0 records E0330 14:59:47.787132 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0330 14:59:47.787138 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 14.649205ms to process 0 records I0330 14:59:47.787142 1 tasks_processing.go:74] worker 38 stopped. I0330 14:59:47.787163 1 tasks_processing.go:74] worker 50 stopped. E0330 14:59:47.787172 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0330 14:59:47.787179 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 13.802008ms to process 0 records E0330 14:59:47.787186 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0330 14:59:47.787191 1 gather.go:177] gatherer "clusterconfig" function "machines" took 14.732515ms to process 0 records I0330 14:59:47.787196 1 tasks_processing.go:74] worker 36 stopped. I0330 14:59:47.797107 1 tasks_processing.go:74] worker 20 stopped. I0330 14:59:47.797129 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 24.399555ms to process 0 records I0330 14:59:47.797137 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 24.145125ms to process 0 records I0330 14:59:47.797144 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 24.356536ms to process 0 records I0330 14:59:47.797149 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 24.196489ms to process 0 records I0330 14:59:47.797155 1 tasks_processing.go:74] worker 16 stopped. I0330 14:59:47.797160 1 tasks_processing.go:74] worker 10 stopped. I0330 14:59:47.797164 1 tasks_processing.go:74] worker 32 stopped. I0330 14:59:47.797246 1 tasks_processing.go:74] worker 6 stopped. I0330 14:59:47.797256 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.33197ms to process 0 records I0330 14:59:47.797263 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 22.092968ms to process 0 records I0330 14:59:47.797269 1 tasks_processing.go:74] worker 14 stopped. E0330 14:59:47.797281 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0330 14:59:47.797293 1 tasks_processing.go:74] worker 26 stopped. I0330 14:59:47.797301 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 22.075786ms to process 0 records I0330 14:59:47.797749 1 tasks_processing.go:74] worker 27 stopped. I0330 14:59:47.798091 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=4fb40c2e4a9e18eaec497cb76b2c8af3d016f53f1adca936f1969070e059d069 I0330 14:59:47.798106 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 24.095767ms to process 1 records I0330 14:59:47.800164 1 gather_logs.go:145] no pods in namespace were found I0330 14:59:47.800177 1 tasks_processing.go:74] worker 29 stopped. I0330 14:59:47.800184 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 27.34902ms to process 0 records I0330 14:59:47.800308 1 tasks_processing.go:74] worker 18 stopped. I0330 14:59:47.800321 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 27.78056ms to process 0 records I0330 14:59:47.800344 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 27.317857ms to process 0 records I0330 14:59:47.800352 1 tasks_processing.go:74] worker 31 stopped. I0330 14:59:47.800390 1 tasks_processing.go:74] worker 35 stopped. I0330 14:59:47.800411 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 27.610902ms to process 0 records I0330 14:59:47.800497 1 tasks_processing.go:74] worker 60 stopped. I0330 14:59:47.800587 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=1d0e3da9bf2876b561f62086f874919c8d9f05a8fa527838d01cb81ed9a09df5 I0330 14:59:47.800610 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=1cd0e367560d826039fad775c8c31faa79dc69ab3b4e18509d4c83ca57c4a2c8 I0330 14:59:47.800624 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=f9b7e3ee2b143984eb5e89af54b4882d8928f21272ba415fdfe478f00830f5c1 I0330 14:59:47.800632 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 28.048049ms to process 3 records I0330 14:59:47.800637 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 27.39354ms to process 0 records I0330 14:59:47.800643 1 tasks_processing.go:74] worker 59 stopped. I0330 14:59:47.800686 1 tasks_processing.go:74] worker 2 stopped. I0330 14:59:47.800900 1 recorder.go:75] Recording config/authentication with fingerprint=ea84c4e7166b4fcce7505e43fcbb4a83dcc3a40bac77a30afe681377f91a37f7 I0330 14:59:47.800919 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 27.665265ms to process 1 records I0330 14:59:47.800928 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 26.776437ms to process 0 records E0330 14:59:47.800937 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0330 14:59:47.800939 1 tasks_processing.go:74] worker 45 stopped. I0330 14:59:47.800944 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 27.298538ms to process 0 records I0330 14:59:47.800945 1 tasks_processing.go:74] worker 54 stopped. I0330 14:59:47.800951 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 26.916561ms to process 0 records I0330 14:59:47.800959 1 tasks_processing.go:74] worker 5 stopped. I0330 14:59:47.801126 1 tasks_processing.go:74] worker 48 stopped. I0330 14:59:47.801144 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 26.937938ms to process 0 records I0330 14:59:47.801360 1 tasks_processing.go:74] worker 52 stopped. I0330 14:59:47.801371 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 26.782193ms to process 0 records I0330 14:59:47.801406 1 tasks_processing.go:74] worker 1 stopped. I0330 14:59:47.801484 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=f1028e5613c1d6660026469caabd700d998f5720b358551539081a0398afe473 I0330 14:59:47.801497 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 28.88314ms to process 1 records I0330 14:59:47.801511 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 27.391448ms to process 0 records I0330 14:59:47.801527 1 tasks_processing.go:74] worker 44 stopped. I0330 14:59:47.801552 1 recorder.go:75] Recording config/proxy with fingerprint=2c7016ec73e4f2270058743647adcf807cd9068be46dbb0f906eec2b69610b7b I0330 14:59:47.801562 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 27.060246ms to process 1 records I0330 14:59:47.801571 1 tasks_processing.go:74] worker 3 stopped. W0330 14:59:47.801610 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0330 14:59:47.801613 1 tasks_processing.go:74] worker 61 stopped. I0330 14:59:47.801684 1 recorder.go:75] Recording config/image with fingerprint=3b856d3a651a901eda82632e24d9b8a178ab61da8a0087d3af832e4879b89ff2 I0330 14:59:47.801704 1 gather.go:177] gatherer "clusterconfig" function "image" took 27.854123ms to process 1 records I0330 14:59:47.801785 1 tasks_processing.go:74] worker 55 stopped. I0330 14:59:47.801821 1 recorder.go:75] Recording config/network with fingerprint=7857184eb33af40d52ceeebdca23fb45862d0895862b37f01525f917b6f855aa I0330 14:59:47.801832 1 gather.go:177] gatherer "clusterconfig" function "networks" took 28.235777ms to process 1 records I0330 14:59:47.802924 1 tasks_processing.go:74] worker 23 stopped. I0330 14:59:47.803698 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=e8a44dde64f6e4b1bbdfce71015288627cd685b6bb632ad8263f93ef7d69857a I0330 14:59:47.803763 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=c521957fb3443c5c9f5ee6b1a66842e19fc113a22ec28a629b7ab95ee006adcc I0330 14:59:47.803809 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=a2fc019607a7a5565387e8254339676f969a59ecadd51d3909fa3b8c933292fe I0330 14:59:47.803821 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 30.184385ms to process 3 records I0330 14:59:47.805189 1 tasks_processing.go:74] worker 37 stopped. I0330 14:59:47.805382 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=04a3364d9ee9e493c40354272bceb43df6a94004b88ea58cd1cfc086c56384a3 I0330 14:59:47.805479 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2f80bd8ce9f4a63bcc93e4cf9e1b5cf667df15d879672aff434dfddbfb53b822 I0330 14:59:47.805490 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 32.757419ms to process 2 records I0330 14:59:47.805593 1 tasks_processing.go:74] worker 46 stopped. I0330 14:59:47.805601 1 recorder.go:75] Recording config/featuregate with fingerprint=bbb7ef6aa99486ea798d0077d4680916b3705717c9467501d453dcaba1accf33 I0330 14:59:47.805611 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 31.365944ms to process 1 records E0330 14:59:47.805624 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0330 14:59:47.805658 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0330 14:59:47.805667 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 31.335665ms to process 1 records I0330 14:59:47.805673 1 tasks_processing.go:74] worker 63 stopped. I0330 14:59:47.805749 1 tasks_processing.go:74] worker 47 stopped. I0330 14:59:47.806056 1 recorder.go:75] Recording config/node/ip-10-0-0-65.ec2.internal with fingerprint=493a01fb2a856c31ff5414b4c3b8f4067b68d139a65fc2501fc4ceac70211fe3 I0330 14:59:47.806130 1 recorder.go:75] Recording config/node/ip-10-0-1-27.ec2.internal with fingerprint=108a6040eb52d575fa18acba021d929e83cabaf7f93b9e7e7c430574453242f3 I0330 14:59:47.806192 1 recorder.go:75] Recording config/node/ip-10-0-2-167.ec2.internal with fingerprint=20ad482b01b06ffce8a5cf55b14e2d1c0f3ce43b4475dce00c4fc2c08a12691c I0330 14:59:47.806201 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 31.19994ms to process 3 records I0330 14:59:47.806391 1 tasks_processing.go:74] worker 62 stopped. I0330 14:59:47.806576 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=3b25b9485782cf410689e2964335f09d41dfd2b59c1c49d0c726eaa97377cb71 I0330 14:59:47.806697 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=5123e5e3f4a93d9e286d528257bb6dc087039f2fa582ccbf2a28f69708235f9a I0330 14:59:47.806723 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=954b382a58878f2223761d6932c5c607fdcb2ea4554465707b38430326c00944 I0330 14:59:47.806760 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=9f05abc0e57404d189e6147712e07899ad3d389291af9f8108474b45053f4472 I0330 14:59:47.806790 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=7757ac32f992c6b9e04e398ebe8144b7cd7adb33911d81cd8f75e92786eae884 I0330 14:59:47.806851 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=2225421f845cd0937150a34d100f9ee4b5ed776edd44f31854ee5cb72f6c5b29 I0330 14:59:47.806881 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=db15667bfc9c1946a906f9a16eacec7af747d45054644d772929e30cc7628e88 I0330 14:59:47.806925 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=77dae220d56dac149a886c32396e6bd83f9d69a8d6ca957b45c71b34c4e42c7b I0330 14:59:47.806967 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=743618f7f7742d4ba119b001af6f8858a867614527ef2aaee07857daf2c4f84e I0330 14:59:47.807008 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=60a86aa241e794b3d1db6516ae9b53d034982755bb7686ca886251f87540756d I0330 14:59:47.807039 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7f4a7a3a3f4b3d3cfb11fd8532fb9f39190b6114f5b573dfcea711f082f9c192 I0330 14:59:47.807051 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 32.629677ms to process 11 records I0330 14:59:47.807072 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0330 14:59:47.807082 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 33.991886ms to process 1 records I0330 14:59:47.807091 1 tasks_processing.go:74] worker 33 stopped. I0330 14:59:47.811117 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0330 14:59:47.811142 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0330 14:59:47.811882 1 operator.go:328] started I0330 14:59:47.811900 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0330 14:59:47.817590 1 tasks_processing.go:74] worker 4 stopped. I0330 14:59:47.817608 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 43.406674ms to process 0 records I0330 14:59:47.817685 1 tasks_processing.go:74] worker 11 stopped. I0330 14:59:47.819936 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-5mknx with fingerprint=c0f06f186cd9ed691ee71f6c9dc245739701693fd58ccd79f80fd7d04eeb47f0 I0330 14:59:47.820038 1 recorder.go:75] Recording config/running_containers with fingerprint=790c058db12c534906c725f628915808380e50687aaa765c04ece43af1463f62 I0330 14:59:47.820048 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 45.237879ms to process 2 records I0330 14:59:47.822810 1 tasks_processing.go:74] worker 49 stopped. I0330 14:59:47.822829 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 50.365473ms to process 0 records I0330 14:59:47.823003 1 tasks_processing.go:74] worker 12 stopped. I0330 14:59:47.823547 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=a8287871a9cc57f70e7077066393511f715ac7853de15d40f0f2306b36dcb2c4 I0330 14:59:47.823740 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=e9ae16ff45a62e696520f7697f1f0c27fb4c0610a7874b43ca86ec38e3a21cea I0330 14:59:47.823749 1 gather.go:177] gatherer "clusterconfig" function "crds" took 47.996859ms to process 2 records I0330 14:59:47.826133 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0330 14:59:47.826147 1 controller.go:254] Source scaController *sca.Controller is not ready I0330 14:59:47.826151 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0330 14:59:47.826154 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0330 14:59:47.826157 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0330 14:59:47.826172 1 controller.go:531] The operator is still being initialized I0330 14:59:47.826179 1 controller.go:554] The operator is healthy E0330 14:59:47.832270 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27e8eb7593-a2e1-4582-97ca-eed5bdf52a32%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40121->172.30.0.10:53: read: connection refused I0330 14:59:47.832282 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27e8eb7593-a2e1-4582-97ca-eed5bdf52a32%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40121->172.30.0.10:53: read: connection refused I0330 14:59:47.833074 1 prometheus_rules.go:88] Prometheus rules successfully created I0330 14:59:47.833947 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0330 14:59:47.833958 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0330 14:59:47.833965 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0330 14:59:47.838144 1 base_controller.go:82] Caches are synced for ConfigController I0330 14:59:47.838157 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0330 14:59:47.865489 1 configmapobserver.go:84] configmaps "insights-config" not found I0330 14:59:47.865510 1 tasks_processing.go:74] worker 25 stopped. I0330 14:59:47.865721 1 recorder.go:75] Recording config/version with fingerprint=39477370818d2dd3e7eec578fa05d39dc7d07215a63af7f6972bd41592d0ab9d I0330 14:59:47.865734 1 recorder.go:75] Recording config/id with fingerprint=3c69bae49e4bad8acc2715872be1ce4ae7bfe0bcecc524517df2d3ca29d22ee8 I0330 14:59:47.865740 1 gather.go:177] gatherer "clusterconfig" function "version" took 92.717194ms to process 2 records I0330 14:59:47.874862 1 tasks_processing.go:74] worker 58 stopped. E0330 14:59:47.874875 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0330 14:59:47.874882 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0330 14:59:47.874886 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0330 14:59:47.874895 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=7c953bef0cdbddb25a47a3c40772dfa7cbf73ae2ac9bb3dc97984dbf1bd43499 I0330 14:59:47.874938 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0330 14:59:47.874946 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0330 14:59:47.874950 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=4bec6f0bfa45f0ccf5709371166150dcb9cf2703d04f5f6de4476d0277c477a6 I0330 14:59:47.874954 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0330 14:59:47.874992 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0330 14:59:47.874999 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0330 14:59:47.875005 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 101.613159ms to process 7 records I0330 14:59:47.912068 1 base_controller.go:82] Caches are synced for LoggingSyncer I0330 14:59:47.912080 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0330 14:59:47.920808 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0330 14:59:47.924502 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:58148->172.30.0.10:53: read: connection refused I0330 14:59:47.924515 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:58148->172.30.0.10:53: read: connection refused I0330 14:59:47.932880 1 tasks_processing.go:74] worker 19 stopped. E0330 14:59:47.932943 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0330 14:59:47.932960 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pc3srqfa75va1b7r65hvfp4e15g46er-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pc3srqfa75va1b7r65hvfp4e15g46er-primary-cert-bundle-secret" not found I0330 14:59:47.933045 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=4116c4b457b4b1eab45d4688e4c5e8d42cce4f856339cfa7f84c48eb67898404 I0330 14:59:47.933061 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 157.742992ms to process 1 records I0330 14:59:47.940274 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0330 14:59:47.944411 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0330 14:59:48.216442 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0330 14:59:48.216456 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0330 14:59:48.217429 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-tctfk pod in namespace openshift-dns (previous: false). I0330 14:59:48.438572 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-tctfk pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-tctfk\" is waiting to start: ContainerCreating" I0330 14:59:48.438589 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-tctfk\" is waiting to start: ContainerCreating" I0330 14:59:48.438598 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-tctfk pod in namespace openshift-dns (previous: false). I0330 14:59:48.623163 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-tctfk pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-tctfk\" is waiting to start: ContainerCreating" I0330 14:59:48.623179 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-tctfk\" is waiting to start: ContainerCreating" I0330 14:59:48.623192 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-458lx pod in namespace openshift-dns (previous: false). W0330 14:59:48.801026 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0330 14:59:48.836621 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:48.836635 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-5jjf9 pod in namespace openshift-dns (previous: false). I0330 14:59:49.024353 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:49.024370 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-jstzh pod in namespace openshift-dns (previous: false). I0330 14:59:49.220095 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:49.220113 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-84884f658b-pb2kz pod in namespace openshift-image-registry (previous: false). I0330 14:59:49.232470 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0330 14:59:49.416459 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:49.416483 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-84884f658b-w459r pod in namespace openshift-image-registry (previous: false). I0330 14:59:49.616083 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:49.616101 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-8547d789cc-z8sdp pod in namespace openshift-image-registry (previous: false). W0330 14:59:49.801605 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0330 14:59:49.822166 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-8547d789cc-z8sdp pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-8547d789cc-z8sdp\" is waiting to start: ContainerCreating" I0330 14:59:49.822188 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-8547d789cc-z8sdp\" is waiting to start: ContainerCreating" I0330 14:59:49.822198 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-8flvq pod in namespace openshift-image-registry (previous: false). I0330 14:59:49.834974 1 tasks_processing.go:74] worker 13 stopped. I0330 14:59:49.835028 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=68d4a18ed81b8be37f14f91be9a0de338a26cac7c0df088ef33db798fe164170 I0330 14:59:49.835077 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=726b4ed6a9fc2fc3520628824351108d06f29345470bcdaa55fcbe0c03e54c8c I0330 14:59:49.835109 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0330 14:59:49.835135 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=42c9059c79015ec0e8575f86012f31b96744d509da11c4ce760a8b4307c8c39b I0330 14:59:49.835152 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0330 14:59:49.835189 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=546577d549127d697d53769709836db72ab69d2b02091200d35da07f5af7eed3 I0330 14:59:49.835220 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=8d8f36bda078f016c60ba0a0eda140a2ceea3c38a6a313ecff92a988195f62bf I0330 14:59:49.835244 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=8ae01ff324a8026fb09b7c95ed41e6c252926a33abb28a334e8fa2e881e0d265 I0330 14:59:49.835258 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=ca92fb6dd50a328879a5e426948be986f93f78d6f78d4d938df2c5c6d51e8249 I0330 14:59:49.835283 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=61598ae55b507704b3c2ad64d0a039d8cced782f825d597a099c2c2617b2cd04 I0330 14:59:49.835293 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0330 14:59:49.835309 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=d63a3c4a5cd85cf5dc995b352ff83b24ca63d21338b849d46103c033c8113ef2 I0330 14:59:49.835320 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0330 14:59:49.835361 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=e5e4410ac49f194a33e43d7c9d61fcccbc20346057ec7de4ddcb3a72600c2675 I0330 14:59:49.835375 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0330 14:59:49.835391 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=90a16002fa5062979f32abfaf38e24c9d37c3642c6c1bfa792f26396565ce16e I0330 14:59:49.835401 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0330 14:59:49.835416 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=755fffc9746fbcf14789cc684159ccc77d7f334d66dbbf13e2ab826b312d265d I0330 14:59:49.835538 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=6d05c3d22d73795857bb423cc268206bb3ccb72b234da14bed65d4c1d8f1d701 I0330 14:59:49.835547 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0330 14:59:49.835554 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0330 14:59:49.835574 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0330 14:59:49.835598 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=56add3211655035e4dc87504c0bb4a25c421501446137f317d4ef930f8a70297 I0330 14:59:49.835621 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=bbd2a393d31c2565107a2aa16c37b3a6c01d7106031ad49762e16d39911710c5 I0330 14:59:49.835630 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0330 14:59:49.835646 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=f26d899379977da5b48f727925a5bcdb97f9563c0b05424fdf5f52106b4a54ff I0330 14:59:49.835656 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0330 14:59:49.835669 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=f5353f4be8527004dd8e5fe9f54233b5dfaae5dead122bca69b90b1a7906a8c2 I0330 14:59:49.835682 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=4cbeab808901f0e5d1a58119000dc6db789405d61858d5807b086ad3d2854264 I0330 14:59:49.835698 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=72aed00a7e4e29467992fbaae51dcfd273bdead78468382c5e09774cb96a3099 I0330 14:59:49.835711 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=6c5fb3612cb8f3adf4563cba0d9405c37f0ca43ec3cbc463517f07e8a175d8da I0330 14:59:49.835732 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=f4f86a45e2d4b609cf9bbeafec3092af02af50c72e6c8739efd189ea78eb6578 I0330 14:59:49.835739 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0330 14:59:49.835767 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=ffc41a55699b9a50b768327d4f3167da90534b1e557b1b1e1a699f1142b20b36 I0330 14:59:49.835784 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0330 14:59:49.835791 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0330 14:59:49.835798 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.061912616s to process 36 records I0330 14:59:50.020243 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:50.020261 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-kr59t pod in namespace openshift-image-registry (previous: false). I0330 14:59:50.221777 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:50.221794 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-v2h97 pod in namespace openshift-image-registry (previous: false). I0330 14:59:50.421478 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0330 14:59:50.421494 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-78459bd87c-6zkgf pod in namespace openshift-ingress (previous: false). I0330 14:59:50.621204 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-78459bd87c-6zkgf pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-78459bd87c-6zkgf\" is waiting to start: ContainerCreating" I0330 14:59:50.621221 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-78459bd87c-6zkgf\" is waiting to start: ContainerCreating" I0330 14:59:50.621233 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-78459bd87c-rz9gd pod in namespace openshift-ingress (previous: false). W0330 14:59:50.800999 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0330 14:59:50.822729 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-78459bd87c-rz9gd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-78459bd87c-rz9gd\" is waiting to start: ContainerCreating" I0330 14:59:50.822748 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-78459bd87c-rz9gd\" is waiting to start: ContainerCreating" I0330 14:59:50.822759 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-79cb556bcf-kl4ns pod in namespace openshift-ingress (previous: false). I0330 14:59:51.032805 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-79cb556bcf-kl4ns pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-79cb556bcf-kl4ns\" is waiting to start: ContainerCreating" I0330 14:59:51.032835 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-79cb556bcf-kl4ns\" is waiting to start: ContainerCreating" I0330 14:59:51.032849 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-qkcsd pod in namespace openshift-ingress-canary (previous: false). I0330 14:59:51.225443 1 tasks_processing.go:74] worker 28 stopped. I0330 14:59:51.225687 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=ec681e3fb45403687b3e6dfc29c1313a2f1a4c667f2db5c00311550677d4e1fd I0330 14:59:51.225738 1 recorder.go:75] Recording events/openshift-dns with fingerprint=59b87f8813a86f620b654ffbf68217a556cea9bbeab164508c629cdbca365aa7 I0330 14:59:51.225999 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=0c2316737814dd2e15ff569b1993a9c5e9647700621aa3835d91e50ac22b7a43 I0330 14:59:51.226115 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=4e24f82697e69ff9c91ab4b290d043cf4250d59ace2794ad7dcc3191dc439053 I0330 14:59:51.226216 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=0052ec725fa7ac75583e84e6360b5e03f80d80998334227877cfe5bb2a12210a I0330 14:59:51.226266 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=96f95c4676dc9fc6878633409724e28323dfbb41ec8dde3203cd25dbbe3eb802 I0330 14:59:51.226293 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-qkcsd/serve-healthcheck-canary_current.log with fingerprint=039710188583bd1157400a41440e11844f0c431eafcbf89f51934a165bde408b I0330 14:59:51.226322 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.450747607s to process 7 records W0330 14:59:51.801603 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0330 14:59:52.801691 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0330 14:59:52.801718 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0330 14:59:52.801733 1 tasks_processing.go:74] worker 51 stopped. E0330 14:59:52.801743 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0330 14:59:52.801755 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0330 14:59:52.801771 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0330 14:59:52.801780 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.028092032s to process 1 records I0330 15:00:00.213523 1 tasks_processing.go:74] worker 22 stopped. I0330 15:00:00.213561 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0330 15:00:00.213576 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.441070125s to process 1 records I0330 15:00:00.726779 1 configmapobserver.go:84] configmaps "insights-config" not found I0330 15:00:00.981401 1 tasks_processing.go:74] worker 56 stopped. I0330 15:00:00.981684 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=7c1a3e3c431de69ce4ee6fb33aad2158a3d38446a91ec5899ca71ceaf725097e I0330 15:00:00.981702 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.206513675s to process 1 records E0330 15:00:00.981760 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.209s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0330 15:00:00.982861 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0330 15:00:00.982876 1 periodic.go:216] Running workloads gatherer I0330 15:00:00.982890 1 tasks_processing.go:45] number of workers: 2 I0330 15:00:00.982901 1 tasks_processing.go:69] worker 1 listening for tasks. I0330 15:00:00.982909 1 tasks_processing.go:69] worker 0 listening for tasks. I0330 15:00:00.982910 1 tasks_processing.go:71] worker 1 working on workload_info task. I0330 15:00:00.982916 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0330 15:00:01.006457 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0330 15:00:01.007802 1 tasks_processing.go:74] worker 0 stopped. I0330 15:00:01.007817 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 24.751164ms to process 0 records I0330 15:00:01.014545 1 gather_workloads_info.go:387] No image sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad (9ms) I0330 15:00:01.022456 1 gather_workloads_info.go:387] No image sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1 (8ms) I0330 15:00:01.030130 1 gather_workloads_info.go:387] No image sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e (8ms) I0330 15:00:01.038056 1 gather_workloads_info.go:387] No image sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda (8ms) I0330 15:00:01.046080 1 gather_workloads_info.go:387] No image sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473 (8ms) I0330 15:00:01.053777 1 gather_workloads_info.go:387] No image sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0 (8ms) I0330 15:00:01.061456 1 gather_workloads_info.go:387] No image sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0 (8ms) I0330 15:00:01.068851 1 gather_workloads_info.go:387] No image sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897 (7ms) I0330 15:00:01.076545 1 gather_workloads_info.go:387] No image sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365 (8ms) I0330 15:00:01.084674 1 gather_workloads_info.go:387] No image sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c (8ms) I0330 15:00:01.115851 1 gather_workloads_info.go:387] No image sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d (31ms) I0330 15:00:01.145441 1 configmapobserver.go:84] configmaps "insights-config" not found I0330 15:00:01.215705 1 gather_workloads_info.go:387] No image sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707 (100ms) I0330 15:00:01.315540 1 gather_workloads_info.go:387] No image sha256:8f72efd51f564d8d3e4ecee6555d7bd5394d0776054d2112febea6b7c9e71225 (100ms) I0330 15:00:01.345271 1 configmapobserver.go:84] configmaps "insights-config" not found I0330 15:00:01.416179 1 gather_workloads_info.go:387] No image sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a (101ms) I0330 15:00:01.516609 1 gather_workloads_info.go:387] No image sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704 (100ms) I0330 15:00:01.615915 1 gather_workloads_info.go:387] No image sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c (99ms) I0330 15:00:01.715498 1 gather_workloads_info.go:387] No image sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6 (100ms) I0330 15:00:01.816042 1 gather_workloads_info.go:387] No image sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2 (101ms) I0330 15:00:01.916111 1 gather_workloads_info.go:387] No image sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4 (100ms) I0330 15:00:02.016947 1 gather_workloads_info.go:387] No image sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40 (101ms) I0330 15:00:02.115528 1 gather_workloads_info.go:387] No image sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950 (99ms) I0330 15:00:02.217441 1 gather_workloads_info.go:387] No image sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137 (102ms) I0330 15:00:02.314781 1 gather_workloads_info.go:387] No image sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56 (97ms) I0330 15:00:02.415066 1 gather_workloads_info.go:387] No image sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6 (100ms) I0330 15:00:02.515178 1 gather_workloads_info.go:387] No image sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8 (100ms) I0330 15:00:02.615548 1 gather_workloads_info.go:387] No image sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc (100ms) I0330 15:00:02.715689 1 gather_workloads_info.go:387] No image sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd (100ms) I0330 15:00:02.815467 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0330 15:00:02.915480 1 gather_workloads_info.go:387] No image sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b (100ms) I0330 15:00:03.016058 1 gather_workloads_info.go:387] No image sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04 (101ms) I0330 15:00:03.115647 1 gather_workloads_info.go:387] No image sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72 (100ms) I0330 15:00:03.215418 1 gather_workloads_info.go:387] No image sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0 (100ms) I0330 15:00:03.315045 1 gather_workloads_info.go:387] No image sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0 (100ms) I0330 15:00:03.415604 1 gather_workloads_info.go:387] No image sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c (101ms) I0330 15:00:03.415640 1 tasks_processing.go:74] worker 1 stopped. E0330 15:00:03.415650 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0330 15:00:03.415919 1 recorder.go:75] Recording config/workload_info with fingerprint=34804e8f420c8bf25c07be3d70f9e054a6cd10b3171b080f99082257297e6ace I0330 15:00:03.415936 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.43271018s to process 1 records E0330 15:00:03.415969 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.433s with: function \"workload_info\" failed with an error" I0330 15:00:03.417077 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0330 15:00:03.417092 1 periodic.go:216] Running conditional gatherer I0330 15:00:03.422920 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.6/gathering_rules I0330 15:00:03.429533 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.6/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:45691->172.30.0.10:53: read: connection refused E0330 15:00:03.429832 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0330 15:00:03.429889 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0330 15:00:03.437341 1 conditional_gatherer.go:392] cluster version is '4.21.6' E0330 15:00:03.437353 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437358 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437362 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437365 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437369 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437378 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437381 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437384 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0330 15:00:03.437387 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0330 15:00:03.437403 1 tasks_processing.go:45] number of workers: 3 I0330 15:00:03.437414 1 tasks_processing.go:69] worker 2 listening for tasks. I0330 15:00:03.437418 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0330 15:00:03.437425 1 tasks_processing.go:69] worker 0 listening for tasks. I0330 15:00:03.437437 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0330 15:00:03.437442 1 tasks_processing.go:69] worker 1 listening for tasks. I0330 15:00:03.437447 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0330 15:00:03.437450 1 tasks_processing.go:74] worker 1 stopped. I0330 15:00:03.437503 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0330 15:00:03.437519 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.327µs to process 1 records I0330 15:00:03.437555 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0330 15:00:03.437568 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.697µs to process 1 records I0330 15:00:03.437582 1 tasks_processing.go:74] worker 0 stopped. I0330 15:00:03.437700 1 tasks_processing.go:74] worker 2 stopped. I0330 15:00:03.437712 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 247.86µs to process 0 records I0330 15:00:03.437740 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.6/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:45691->172.30.0.10:53: read: connection refused I0330 15:00:03.437760 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0330 15:00:03.461900 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=6f3eb6615e606fb17db9ff59ffb54c27da7a4ad71840643fa553d8c2514d25e3 I0330 15:00:03.462020 1 diskrecorder.go:70] Writing 102 records to /var/lib/insights-operator/insights-2026-03-30-150003.tar.gz I0330 15:00:03.468222 1 diskrecorder.go:51] Wrote 102 records to disk in 6ms I0330 15:00:03.468258 1 periodic.go:285] Gathering cluster info every 2h0m0s I0330 15:00:03.468283 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0330 15:00:15.379964 1 configmapobserver.go:84] configmaps "insights-config" not found I0330 15:01:01.966240 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="53c041ebfed7e9f90f758769fea3303ed377aee8e727e123189f7f0554cedf67") W0330 15:01:01.966273 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0330 15:01:01.966319 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="b389644c33bb967657c1f56711130858a36d8a201c330e71ee61c5249764702c")