W0424 02:56:55.630094 1 cmd.go:257] Using insecure, self-signed certificates I0424 02:56:56.003977 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:56:56.004305 1 observer_polling.go:159] Starting file observer I0424 02:56:56.553541 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0424 02:56:56.553730 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0424 02:56:56.554282 1 secure_serving.go:57] Forcing use of http/1.1 only W0424 02:56:56.554306 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0424 02:56:56.554310 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0424 02:56:56.554315 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0424 02:56:56.554318 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0424 02:56:56.554320 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0424 02:56:56.554322 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0424 02:56:56.554284 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0424 02:56:56.558825 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0424 02:56:56.558830 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"74a2fb29-dfa5-405d-a82f-0fa9e3bcc786", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0424 02:56:56.559731 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0424 02:56:56.559740 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0424 02:56:56.559749 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0424 02:56:56.559754 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 02:56:56.559762 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0424 02:56:56.559779 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 02:56:56.560020 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3076153006/tls.crt::/tmp/serving-cert-3076153006/tls.key" I0424 02:56:56.560275 1 secure_serving.go:213] Serving securely on [::]:8443 I0424 02:56:56.560310 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0424 02:56:56.564529 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0424 02:56:56.564567 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 02:56:56.564656 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0424 02:56:56.570277 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 02:56:56.570351 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 02:56:56.576066 1 secretconfigobserver.go:119] support secret does not exist I0424 02:56:56.581458 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 02:56:56.588198 1 secretconfigobserver.go:119] support secret does not exist I0424 02:56:56.589949 1 recorder.go:176] Pruning old reports every 5h56m49s, max age is 288h0m0s I0424 02:56:56.596786 1 periodic.go:216] Running clusterconfig gatherer I0424 02:56:56.596789 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0424 02:56:56.596813 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0424 02:56:56.596838 1 tasks_processing.go:45] number of workers: 64 I0424 02:56:56.596849 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0424 02:56:56.596855 1 insightsreport.go:296] Starting report retriever I0424 02:56:56.596862 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0424 02:56:56.596862 1 tasks_processing.go:69] worker 6 listening for tasks. I0424 02:56:56.596863 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:56:56.596887 1 tasks_processing.go:69] worker 8 listening for tasks. I0424 02:56:56.596876 1 tasks_processing.go:69] worker 4 listening for tasks. I0424 02:56:56.596878 1 tasks_processing.go:69] worker 5 listening for tasks. I0424 02:56:56.596889 1 tasks_processing.go:69] worker 7 listening for tasks. I0424 02:56:56.596897 1 tasks_processing.go:69] worker 11 listening for tasks. I0424 02:56:56.596908 1 tasks_processing.go:69] worker 10 listening for tasks. I0424 02:56:56.596909 1 tasks_processing.go:69] worker 9 listening for tasks. I0424 02:56:56.596909 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 02:56:56.596915 1 tasks_processing.go:69] worker 16 listening for tasks. I0424 02:56:56.596920 1 tasks_processing.go:69] worker 14 listening for tasks. I0424 02:56:56.596921 1 tasks_processing.go:69] worker 15 listening for tasks. I0424 02:56:56.596892 1 tasks_processing.go:69] worker 3 listening for tasks. I0424 02:56:56.596927 1 tasks_processing.go:71] worker 14 working on machine_healthchecks task. I0424 02:56:56.596951 1 tasks_processing.go:69] worker 17 listening for tasks. I0424 02:56:56.596953 1 tasks_processing.go:69] worker 22 listening for tasks. I0424 02:56:56.596956 1 tasks_processing.go:69] worker 23 listening for tasks. I0424 02:56:56.596952 1 tasks_processing.go:69] worker 40 listening for tasks. I0424 02:56:56.596962 1 tasks_processing.go:69] worker 41 listening for tasks. I0424 02:56:56.596957 1 tasks_processing.go:69] worker 39 listening for tasks. I0424 02:56:56.596969 1 tasks_processing.go:69] worker 19 listening for tasks. I0424 02:56:56.596972 1 tasks_processing.go:69] worker 34 listening for tasks. I0424 02:56:56.596975 1 tasks_processing.go:69] worker 42 listening for tasks. I0424 02:56:56.596974 1 tasks_processing.go:71] worker 41 working on certificate_signing_requests task. I0424 02:56:56.596979 1 tasks_processing.go:71] worker 39 working on sap_datahubs task. I0424 02:56:56.596981 1 tasks_processing.go:69] worker 35 listening for tasks. I0424 02:56:56.596986 1 tasks_processing.go:69] worker 30 listening for tasks. I0424 02:56:56.596987 1 tasks_processing.go:69] worker 36 listening for tasks. I0424 02:56:56.596977 1 tasks_processing.go:69] worker 18 listening for tasks. I0424 02:56:56.596994 1 tasks_processing.go:71] worker 8 working on openshift_machine_api_events task. I0424 02:56:56.596997 1 tasks_processing.go:69] worker 38 listening for tasks. I0424 02:56:56.596987 1 tasks_processing.go:71] worker 3 working on proxies task. I0424 02:56:56.597002 1 tasks_processing.go:71] worker 17 working on config_maps task. I0424 02:56:56.597004 1 tasks_processing.go:71] worker 11 working on version task. I0424 02:56:56.597009 1 tasks_processing.go:69] worker 33 listening for tasks. I0424 02:56:56.597011 1 tasks_processing.go:69] worker 43 listening for tasks. I0424 02:56:56.596908 1 tasks_processing.go:69] worker 12 listening for tasks. I0424 02:56:56.596945 1 tasks_processing.go:69] worker 21 listening for tasks. I0424 02:56:56.596901 1 tasks_processing.go:69] worker 20 listening for tasks. I0424 02:56:56.596956 1 tasks_processing.go:69] worker 26 listening for tasks. I0424 02:56:56.596964 1 tasks_processing.go:69] worker 27 listening for tasks. I0424 02:56:56.596964 1 tasks_processing.go:69] worker 24 listening for tasks. I0424 02:56:56.597028 1 tasks_processing.go:69] worker 45 listening for tasks. I0424 02:56:56.596924 1 tasks_processing.go:69] worker 29 listening for tasks. I0424 02:56:56.596980 1 tasks_processing.go:69] worker 37 listening for tasks. I0424 02:56:56.597039 1 tasks_processing.go:71] worker 10 working on schedulers task. I0424 02:56:56.597046 1 tasks_processing.go:69] worker 46 listening for tasks. I0424 02:56:56.597054 1 tasks_processing.go:69] worker 47 listening for tasks. I0424 02:56:56.597061 1 tasks_processing.go:69] worker 48 listening for tasks. I0424 02:56:56.597069 1 tasks_processing.go:69] worker 49 listening for tasks. I0424 02:56:56.597079 1 tasks_processing.go:69] worker 50 listening for tasks. I0424 02:56:56.597088 1 tasks_processing.go:69] worker 51 listening for tasks. I0424 02:56:56.597094 1 tasks_processing.go:71] worker 4 working on infrastructures task. I0424 02:56:56.597102 1 tasks_processing.go:69] worker 53 listening for tasks. I0424 02:56:56.597110 1 tasks_processing.go:69] worker 54 listening for tasks. I0424 02:56:56.597117 1 tasks_processing.go:69] worker 55 listening for tasks. I0424 02:56:56.597125 1 tasks_processing.go:69] worker 56 listening for tasks. I0424 02:56:56.597132 1 tasks_processing.go:69] worker 57 listening for tasks. I0424 02:56:56.597139 1 tasks_processing.go:71] worker 2 working on sap_config task. I0424 02:56:56.596945 1 tasks_processing.go:69] worker 25 listening for tasks. I0424 02:56:56.596995 1 tasks_processing.go:69] worker 31 listening for tasks. I0424 02:56:56.597001 1 tasks_processing.go:69] worker 32 listening for tasks. I0424 02:56:56.597022 1 tasks_processing.go:69] worker 44 listening for tasks. I0424 02:56:56.596971 1 tasks_processing.go:69] worker 28 listening for tasks. I0424 02:56:56.597035 1 tasks_processing.go:71] worker 9 working on ingress_certificates task. I0424 02:56:56.596980 1 tasks_processing.go:71] worker 15 working on nodenetworkconfigurationpolicies task. I0424 02:56:56.597238 1 tasks_processing.go:71] worker 28 working on clusterroles task. I0424 02:56:56.597267 1 tasks_processing.go:71] worker 5 working on operators_pods_and_events task. I0424 02:56:56.596901 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:56:56.597341 1 tasks_processing.go:71] worker 1 working on silenced_alerts task. W0424 02:56:56.597364 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:56:56.597378 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 25.881µs to process 0 records I0424 02:56:56.596924 1 tasks_processing.go:71] worker 16 working on openstack_dataplanedeployments task. I0424 02:56:56.597384 1 tasks_processing.go:71] worker 1 working on authentication task. I0424 02:56:56.597403 1 tasks_processing.go:69] worker 59 listening for tasks. I0424 02:56:56.597416 1 tasks_processing.go:71] worker 59 working on operators task. I0424 02:56:56.597446 1 tasks_processing.go:71] worker 22 working on machines task. I0424 02:56:56.597515 1 tasks_processing.go:69] worker 63 listening for tasks. I0424 02:56:56.597526 1 tasks_processing.go:69] worker 60 listening for tasks. I0424 02:56:56.597532 1 tasks_processing.go:69] worker 61 listening for tasks. I0424 02:56:56.597534 1 tasks_processing.go:71] worker 45 working on pdbs task. I0424 02:56:56.597538 1 tasks_processing.go:69] worker 62 listening for tasks. I0424 02:56:56.597542 1 tasks_processing.go:71] worker 38 working on ceph_cluster task. I0424 02:56:56.597615 1 tasks_processing.go:71] worker 23 working on image_pruners task. I0424 02:56:56.597638 1 tasks_processing.go:71] worker 6 working on image_registries task. I0424 02:56:56.597665 1 tasks_processing.go:71] worker 29 working on ingress task. I0424 02:56:56.597675 1 tasks_processing.go:71] worker 33 working on sap_pods task. I0424 02:56:56.597689 1 tasks_processing.go:71] worker 40 working on install_plans task. I0424 02:56:56.597693 1 tasks_processing.go:71] worker 19 working on active_alerts task. W0424 02:56:56.597708 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:56:56.597716 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 14.725µs to process 0 records I0424 02:56:56.597724 1 tasks_processing.go:71] worker 34 working on qemu_kubevirt_launcher_logs task. I0424 02:56:56.596995 1 tasks_processing.go:69] worker 58 listening for tasks. I0424 02:56:56.597765 1 tasks_processing.go:71] worker 20 working on storage_cluster task. I0424 02:56:56.597770 1 tasks_processing.go:71] worker 42 working on feature_gates task. I0424 02:56:56.597776 1 tasks_processing.go:71] worker 26 working on olm_operators task. I0424 02:56:56.597816 1 tasks_processing.go:71] worker 37 working on tsdb_status task. I0424 02:56:56.597873 1 tasks_processing.go:71] worker 35 working on machine_config_pools task. W0424 02:56:56.597891 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:56:56.597906 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 26.869µs to process 0 records I0424 02:56:56.597918 1 tasks_processing.go:71] worker 46 working on aggregated_monitoring_cr_names task. I0424 02:56:56.597920 1 tasks_processing.go:71] worker 24 working on image task. I0424 02:56:56.598021 1 tasks_processing.go:71] worker 30 working on machine_sets task. I0424 02:56:56.598117 1 tasks_processing.go:71] worker 47 working on mutating_webhook_configurations task. I0424 02:56:56.598155 1 tasks_processing.go:71] worker 7 working on service_accounts task. I0424 02:56:56.598288 1 tasks_processing.go:71] worker 12 working on validating_webhook_configurations task. I0424 02:56:56.598381 1 tasks_processing.go:71] worker 36 working on support_secret task. I0424 02:56:56.598414 1 tasks_processing.go:71] worker 21 working on storage_classes task. I0424 02:56:56.598466 1 tasks_processing.go:71] worker 62 working on crds task. I0424 02:56:56.598538 1 tasks_processing.go:71] worker 43 working on machine_autoscalers task. I0424 02:56:56.597096 1 tasks_processing.go:69] worker 52 listening for tasks. I0424 02:56:56.598442 1 tasks_processing.go:71] worker 0 working on nodenetworkstates task. I0424 02:56:56.597771 1 tasks_processing.go:71] worker 27 working on machine_configs task. I0424 02:56:56.598673 1 tasks_processing.go:71] worker 52 working on pod_network_connectivity_checks task. I0424 02:56:56.598690 1 tasks_processing.go:71] worker 32 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0424 02:56:56.596943 1 tasks_processing.go:69] worker 13 listening for tasks. I0424 02:56:56.598459 1 tasks_processing.go:71] worker 60 working on oauths task. I0424 02:56:56.598744 1 tasks_processing.go:71] worker 44 working on openstack_version task. I0424 02:56:56.598757 1 tasks_processing.go:71] worker 31 working on openstack_dataplanenodesets task. I0424 02:56:56.598455 1 tasks_processing.go:71] worker 63 working on nodes task. I0424 02:56:56.598462 1 tasks_processing.go:71] worker 61 working on container_runtime_configs task. I0424 02:56:56.598451 1 tasks_processing.go:71] worker 37 working on networks task. I0424 02:56:56.598575 1 tasks_processing.go:71] worker 58 working on metrics task. W0424 02:56:56.599443 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:56:56.599459 1 tasks_processing.go:74] worker 58 stopped. I0424 02:56:56.599470 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 44.029µs to process 0 records I0424 02:56:56.598580 1 tasks_processing.go:71] worker 55 working on node_features task. I0424 02:56:56.598584 1 tasks_processing.go:71] worker 48 working on overlapping_namespace_uids task. I0424 02:56:56.598589 1 tasks_processing.go:71] worker 49 working on node_logs task. I0424 02:56:56.598592 1 tasks_processing.go:71] worker 50 working on dvo_metrics task. I0424 02:56:56.600067 1 tasks_processing.go:74] worker 13 stopped. I0424 02:56:56.598594 1 tasks_processing.go:71] worker 51 working on container_images task. I0424 02:56:56.598596 1 tasks_processing.go:71] worker 53 working on cost_management_metrics_configs task. I0424 02:56:56.598598 1 tasks_processing.go:71] worker 54 working on cluster_apiserver task. I0424 02:56:56.598601 1 tasks_processing.go:71] worker 25 working on monitoring_persistent_volumes task. I0424 02:56:56.598603 1 tasks_processing.go:71] worker 56 working on openstack_controlplanes task. I0424 02:56:56.598606 1 tasks_processing.go:71] worker 57 working on lokistack task. I0424 02:56:56.598569 1 tasks_processing.go:71] worker 18 working on jaegers task. I0424 02:56:56.598563 1 tasks_processing.go:71] worker 19 working on openshift_logging task. I0424 02:56:56.624645 1 tasks_processing.go:74] worker 15 stopped. I0424 02:56:56.624667 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 27.40899ms to process 0 records I0424 02:56:56.624679 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 27.506396ms to process 0 records E0424 02:56:56.624688 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0424 02:56:56.624699 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.198359ms to process 0 records E0424 02:56:56.624708 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0424 02:56:56.624717 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 27.703452ms to process 0 records I0424 02:56:56.624726 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 26.05153ms to process 0 records I0424 02:56:56.624731 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 25.923173ms to process 0 records I0424 02:56:56.624733 1 tasks_processing.go:74] worker 22 stopped. I0424 02:56:56.624737 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 25.903216ms to process 0 records I0424 02:56:56.624741 1 tasks_processing.go:74] worker 31 stopped. I0424 02:56:56.624742 1 tasks_processing.go:74] worker 44 stopped. I0424 02:56:56.624735 1 tasks_processing.go:74] worker 2 stopped. I0424 02:56:56.624745 1 tasks_processing.go:74] worker 14 stopped. I0424 02:56:56.624744 1 tasks_processing.go:74] worker 43 stopped. I0424 02:56:56.624754 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 27.349474ms to process 0 records I0424 02:56:56.624760 1 tasks_processing.go:74] worker 16 stopped. I0424 02:56:56.624872 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0424 02:56:56.624896 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 02:56:56.624904 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 02:56:56.624910 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 02:56:56.624951 1 controller.go:531] The operator is still being initialized I0424 02:56:56.624964 1 controller.go:554] The operator is healthy I0424 02:56:56.625020 1 tasks_processing.go:74] worker 38 stopped. I0424 02:56:56.625033 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 27.470022ms to process 0 records I0424 02:56:56.625578 1 tasks_processing.go:74] worker 36 stopped. E0424 02:56:56.625589 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0424 02:56:56.625596 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 27.17458ms to process 0 records I0424 02:56:56.626362 1 tasks_processing.go:74] worker 30 stopped. I0424 02:56:56.626374 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 28.328445ms to process 0 records I0424 02:56:56.626381 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 28.477738ms to process 0 records I0424 02:56:56.626385 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 28.690177ms to process 0 records I0424 02:56:56.626391 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 28.600587ms to process 0 records I0424 02:56:56.626398 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 29.40068ms to process 0 records I0424 02:56:56.626410 1 tasks_processing.go:74] worker 39 stopped. I0424 02:56:56.626412 1 tasks_processing.go:74] worker 33 stopped. I0424 02:56:56.626418 1 tasks_processing.go:74] worker 35 stopped. I0424 02:56:56.626418 1 tasks_processing.go:74] worker 20 stopped. I0424 02:56:56.626547 1 tasks_processing.go:74] worker 8 stopped. E0424 02:56:56.626563 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0424 02:56:56.626563 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 29.54518ms to process 0 records I0424 02:56:56.626583 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 27.091372ms to process 0 records I0424 02:56:56.626592 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 27.942816ms to process 0 records I0424 02:56:56.626601 1 tasks_processing.go:74] worker 0 stopped. I0424 02:56:56.626599 1 tasks_processing.go:74] worker 55 stopped. I0424 02:56:56.626608 1 tasks_processing.go:74] worker 10 stopped. I0424 02:56:56.626723 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=96a653efad9478a2aac1f200d71cf96fa6e785b8cf032d6836afd81eecbba6e8 I0424 02:56:56.626739 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 29.559067ms to process 1 records I0424 02:56:56.626782 1 recorder.go:75] Recording config/proxy with fingerprint=f1bba39a4ab4ac070644ff6b8653440ebf3d1881ec0b6bacbadd58cf96547cc9 I0424 02:56:56.626790 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 29.597314ms to process 1 records I0424 02:56:56.626834 1 tasks_processing.go:74] worker 3 stopped. I0424 02:56:56.626902 1 tasks_processing.go:74] worker 23 stopped. I0424 02:56:56.627033 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=8de64488ae59d3d5f2a1d43a23155463c0c1d4080c696468c58f41de366c4387 I0424 02:56:56.627045 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 29.023797ms to process 1 records I0424 02:56:56.627055 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 27.935616ms to process 0 records I0424 02:56:56.627071 1 tasks_processing.go:74] worker 61 stopped. I0424 02:56:56.627148 1 tasks_processing.go:74] worker 4 stopped. I0424 02:56:56.627587 1 recorder.go:75] Recording config/infrastructure with fingerprint=c9a1dc7dabe9de6a99dbc010294a0340b432e50720c9cb960c6bfd7b1ccb822a I0424 02:56:56.627600 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 29.681645ms to process 1 records I0424 02:56:56.627681 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=4da266dba632d176556a3f188f4a91dc4207ddf3a901f84e21af4e8d5bc65299 I0424 02:56:56.627699 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=06b20633e31fe2696934e6170483b11c076afd8fed741d7eb0a68c9b6b87c3a5 I0424 02:56:56.627713 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e565844f5b74461ef49dc503ba9e7ab992d8c57383d4bbdb50c11bbe148a5bd0 I0424 02:56:56.627718 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.288751ms to process 3 records I0424 02:56:56.627725 1 tasks_processing.go:74] worker 45 stopped. I0424 02:56:56.627774 1 recorder.go:75] Recording config/image with fingerprint=e918e262844920ed9d5ac3f344910471d312cad3499703c870ef00135dafd9c4 I0424 02:56:56.627781 1 gather.go:177] gatherer "clusterconfig" function "image" took 29.127051ms to process 1 records I0424 02:56:56.627797 1 tasks_processing.go:74] worker 24 stopped. I0424 02:56:56.627860 1 tasks_processing.go:74] worker 29 stopped. I0424 02:56:56.627867 1 recorder.go:75] Recording config/ingress with fingerprint=5413ab07ed16c0b80aeb418388816e29d5f34899d01de0299504f5942b8bf7b5 I0424 02:56:56.627875 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 29.55874ms to process 1 records I0424 02:56:56.627974 1 tasks_processing.go:74] worker 1 stopped. I0424 02:56:56.628019 1 recorder.go:75] Recording config/authentication with fingerprint=8f0454b775519f7631432cf473cc2c402c5e8fc63028b5b5c051b7b3f79c8d52 I0424 02:56:56.628028 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 29.856687ms to process 1 records E0424 02:56:56.628038 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0424 02:56:56.628044 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 28.986247ms to process 0 records I0424 02:56:56.628050 1 tasks_processing.go:74] worker 52 stopped. I0424 02:56:56.628180 1 gather_logs.go:145] no pods in namespace were found I0424 02:56:56.628196 1 tasks_processing.go:74] worker 42 stopped. I0424 02:56:56.628315 1 recorder.go:75] Recording config/featuregate with fingerprint=fae2ae84154981f88dedc5ae355f8a0488f39a3cf4dc4cfaf7da195b4fe5af65 I0424 02:56:56.628332 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 30.410665ms to process 1 records I0424 02:56:56.628344 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 30.461139ms to process 0 records I0424 02:56:56.628350 1 tasks_processing.go:74] worker 34 stopped. I0424 02:56:56.628854 1 tasks_processing.go:74] worker 6 stopped. I0424 02:56:56.629410 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=8453c15cd7cf48d5e9b1c199e57f371413465dcd8732e34eb32418ee7d114d05 I0424 02:56:56.629467 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 31.2067ms to process 1 records I0424 02:56:56.629767 1 tasks_processing.go:74] worker 18 stopped. I0424 02:56:56.629777 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 28.871954ms to process 0 records I0424 02:56:56.629782 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 29.094086ms to process 0 records I0424 02:56:56.629786 1 tasks_processing.go:74] worker 57 stopped. I0424 02:56:56.630145 1 tasks_processing.go:74] worker 53 stopped. I0424 02:56:56.630163 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 29.905186ms to process 0 records I0424 02:56:56.630573 1 tasks_processing.go:74] worker 41 stopped. I0424 02:56:56.630590 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 33.579356ms to process 0 records I0424 02:56:56.630789 1 tasks_processing.go:74] worker 56 stopped. I0424 02:56:56.630803 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 30.239091ms to process 0 records I0424 02:56:56.630811 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 29.801166ms to process 0 records I0424 02:56:56.630818 1 tasks_processing.go:74] worker 19 stopped. I0424 02:56:56.632050 1 tasks_processing.go:74] worker 60 stopped. I0424 02:56:56.632283 1 recorder.go:75] Recording config/oauth with fingerprint=6eb0dc0e52fa358d9613e161ff25e1419513216d259c6af39acffad13df847a6 I0424 02:56:56.632298 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 33.289823ms to process 1 records I0424 02:56:56.632361 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=cc4bf3ce57ff65fb8e0a4cfb422ae2470d368d75cdd402ef98735f1eceaf73a7 I0424 02:56:56.632380 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=ea81484c18416d53ce866ce7557370f1a58bbec474e96910c5fad87aaf234939 I0424 02:56:56.632386 1 tasks_processing.go:74] worker 21 stopped. I0424 02:56:56.632398 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 33.706094ms to process 2 records I0424 02:56:56.632659 1 tasks_processing.go:74] worker 47 stopped. I0424 02:56:56.632814 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=096baebc1525aa2cda6667daf2147715d9729a28741777dd3791c2e011232394 I0424 02:56:56.632853 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=3c5e49cf7ab7b50e9e036d4995a9fc57a548b6c35b3034e96372610c74f4ccc5 I0424 02:56:56.632879 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=706c17d37e82f8d890f5a302278118dbb8d886e302fe3f0f92384968da9b019a I0424 02:56:56.632888 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 34.525539ms to process 3 records I0424 02:56:56.633208 1 tasks_processing.go:74] worker 27 stopped. E0424 02:56:56.633224 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0424 02:56:56.633252 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0424 02:56:56.633271 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 34.537589ms to process 1 records I0424 02:56:56.633455 1 tasks_processing.go:74] worker 37 stopped. I0424 02:56:56.633577 1 recorder.go:75] Recording config/network with fingerprint=a403e694fbfad318e4ceaf6721785218649bb75f64d538a0246c326099a26c19 I0424 02:56:56.633592 1 gather.go:177] gatherer "clusterconfig" function "networks" took 34.345996ms to process 1 records I0424 02:56:56.636521 1 tasks_processing.go:74] worker 54 stopped. I0424 02:56:56.636691 1 recorder.go:75] Recording config/apiserver with fingerprint=87a51b6a6f030f3d890619ed3c7476a60de2f7601b97d4fddfb9ceea418ff4e4 I0424 02:56:56.636709 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 36.106283ms to process 1 records I0424 02:56:56.636722 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 37.029775ms to process 0 records I0424 02:56:56.636730 1 tasks_processing.go:74] worker 49 stopped. I0424 02:56:56.636757 1 tasks_processing.go:74] worker 25 stopped. I0424 02:56:56.636771 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 36.287011ms to process 0 records I0424 02:56:56.636956 1 tasks_processing.go:74] worker 26 stopped. I0424 02:56:56.636973 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 39.167698ms to process 0 records I0424 02:56:56.638324 1 tasks_processing.go:74] worker 11 stopped. I0424 02:56:56.638562 1 recorder.go:75] Recording config/version with fingerprint=f36756f64ba7572126809f78a07abdcc025089502005b48c9c6032022df8cce8 I0424 02:56:56.638578 1 recorder.go:75] Recording config/id with fingerprint=0caec8f81b7acb5ee3d111b82e8409c2646ff2af5fa9cfb512c8975d0891a948 I0424 02:56:56.638584 1 gather.go:177] gatherer "clusterconfig" function "version" took 41.307106ms to process 2 records I0424 02:56:56.638752 1 tasks_processing.go:74] worker 63 stopped. I0424 02:56:56.638870 1 recorder.go:75] Recording config/node/ip-10-0-0-104.ec2.internal with fingerprint=4c3d95c8e874fa6297158c90abbc407033917fb8ad0cf268f2e563968c229425 I0424 02:56:56.638924 1 recorder.go:75] Recording config/node/ip-10-0-1-139.ec2.internal with fingerprint=9e40db984b753b394a962d30690410ce717f8f215abcb97900529bd9147c9893 I0424 02:56:56.638993 1 recorder.go:75] Recording config/node/ip-10-0-2-137.ec2.internal with fingerprint=44cdd97136906e48f4920d934099ce007cab3c47fbafe074ddacf5a04ad4ac47 I0424 02:56:56.639001 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 39.6464ms to process 3 records I0424 02:56:56.639092 1 tasks_processing.go:74] worker 12 stopped. I0424 02:56:56.639111 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=a7a6248ccedcd48e8b7c4092a836392072f24ad29ce48597cd0fe8957c749e88 I0424 02:56:56.639212 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=10c2296a3f1c54922dbf721c4f310f4704640705bd23bd271f88b79d53d9c83e I0424 02:56:56.639247 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=c5f168ad475393224680bbe64d819effb6b5c6b93bfa73a24c64f62cffbf0f3c I0424 02:56:56.639280 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=523685af50162a71026ef7735256d5389488b04c1da3e154f950d28ce8c48685 I0424 02:56:56.639310 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=b893dd2b98f3cb08579fb6c5026d786ca08a685709f1809c400953d3c149bf26 I0424 02:56:56.639342 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=31a0666d783785ae4dfb5c52b58598e27bd16a6a184a68bce6debe61103d775d I0424 02:56:56.639377 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=f2fa221e09970907fc910fc89ad5513a54630db304978eaf1537f4463aa89c06 I0424 02:56:56.639432 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=014d1ffed780fd200f4d827bd14268eb298cd3b212cda5c4060c585c443a0920 I0424 02:56:56.639458 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=159a43974a15a3b58e7d4594ea22323f0c14c5a19d5aa30e25fb9520b847f77c I0424 02:56:56.639483 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=46ba1a2b431f8d6baf15bef7c98c6d7fd7796397bfb3ff37d19b5a0f1cddaea7 I0424 02:56:56.639749 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=62cf181be135945fc418e6d9bcda8b2018ef2f41c297a2dafd982dccb07ca1ae I0424 02:56:56.639767 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 40.581323ms to process 11 records I0424 02:56:56.639801 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0424 02:56:56.639822 1 tasks_processing.go:74] worker 48 stopped. I0424 02:56:56.639925 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 39.471461ms to process 1 records I0424 02:56:56.640154 1 tasks_processing.go:74] worker 28 stopped. I0424 02:56:56.641154 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=468611db9429f2ecac16dd29197b942ff0f21b0b7e6f6fcecea3a7a8455d0004 I0424 02:56:56.641264 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=d1e59e52aededdef925e6c6e22ceb813485a09d9baf6aa845a02787c2458cf0c I0424 02:56:56.641279 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 42.902116ms to process 2 records I0424 02:56:56.642137 1 tasks_processing.go:74] worker 62 stopped. I0424 02:56:56.642671 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=016467c3b4f6272770449a9b4c5dc30d10c92465d3d0f728450163e01fe7c239 W0424 02:56:56.642758 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:56:56.642892 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=a50a5e80e28cb7c7b3ffbc28b1cf83cd7134492d6d24bcbcfb9a95a52d7a8c6a I0424 02:56:56.642901 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.61556ms to process 2 records I0424 02:56:56.647019 1 tasks_processing.go:74] worker 46 stopped. I0424 02:56:56.647034 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 49.088506ms to process 0 records I0424 02:56:56.653757 1 tasks_processing.go:74] worker 51 stopped. I0424 02:56:56.653826 1 recorder.go:75] Recording config/running_containers with fingerprint=738bce760402705654a37b9d079172ac6fbc8b04930dfc68f03e7838db7d2770 I0424 02:56:56.653838 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 53.569468ms to process 1 records I0424 02:56:56.656027 1 tasks_processing.go:74] worker 32 stopped. I0424 02:56:56.656039 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 57.324082ms to process 0 records I0424 02:56:56.660130 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 02:56:56.660135 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 02:56:56.660157 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0424 02:56:56.661085 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0424 02:56:56.661098 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0424 02:56:56.661249 1 operator.go:328] started I0424 02:56:56.661275 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0424 02:56:56.663390 1 tasks_processing.go:74] worker 17 stopped. E0424 02:56:56.663407 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0424 02:56:56.663415 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0424 02:56:56.663418 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0424 02:56:56.663429 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0424 02:56:56.663464 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0424 02:56:56.663471 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0424 02:56:56.663475 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0424 02:56:56.663480 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0424 02:56:56.663516 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0424 02:56:56.663524 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0424 02:56:56.663529 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 66.372086ms to process 7 records I0424 02:56:56.665454 1 base_controller.go:82] Caches are synced for ConfigController I0424 02:56:56.665467 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0424 02:56:56.675299 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 02:56:56.675310 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 02:56:56.675313 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 02:56:56.675317 1 controller.go:254] Source scaController *sca.Controller is not ready I0424 02:56:56.675320 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0424 02:56:56.675335 1 controller.go:531] The operator is still being initialized I0424 02:56:56.675342 1 controller.go:554] The operator is healthy I0424 02:56:56.682940 1 prometheus_rules.go:88] Prometheus rules successfully created I0424 02:56:56.690437 1 configmapobserver.go:84] configmaps "insights-config" not found E0424 02:56:56.691430 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27ff3c42e6-52a2-4333-9783-b331ebf4570a%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:34406->172.30.0.10:53: read: connection refused I0424 02:56:56.691441 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27ff3c42e6-52a2-4333-9783-b331ebf4570a%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:34406->172.30.0.10:53: read: connection refused I0424 02:56:56.739264 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0424 02:56:56.743109 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:49743->172.30.0.10:53: read: connection refused I0424 02:56:56.743125 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:49743->172.30.0.10:53: read: connection refused I0424 02:56:56.744607 1 tasks_processing.go:74] worker 9 stopped. E0424 02:56:56.744629 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0424 02:56:56.744639 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ps8kd9n54vkadsmhv5dl8r11de9333h-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ps8kd9n54vkadsmhv5dl8r11de9333h-primary-cert-bundle-secret" not found I0424 02:56:56.744707 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=edcaf01304285f190477b9a8377b680c6723b06cdb8229e2c58c4386f1f44a04 I0424 02:56:56.744724 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 147.389945ms to process 1 records I0424 02:56:56.762033 1 base_controller.go:82] Caches are synced for LoggingSyncer I0424 02:56:56.762052 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0424 02:56:57.040125 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0424 02:56:57.040141 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0424 02:56:57.041138 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-5vrvj pod in namespace openshift-dns (previous: false). I0424 02:56:57.265720 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-5vrvj pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-5vrvj\" is waiting to start: ContainerCreating" I0424 02:56:57.265739 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-5vrvj\" is waiting to start: ContainerCreating" I0424 02:56:57.265752 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-5vrvj pod in namespace openshift-dns (previous: false). I0424 02:56:57.452247 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-5vrvj pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-5vrvj\" is waiting to start: ContainerCreating" I0424 02:56:57.452265 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-5vrvj\" is waiting to start: ContainerCreating" I0424 02:56:57.452278 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-n4xfq pod in namespace openshift-dns (previous: false). W0424 02:56:57.641072 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:56:57.673025 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-n4xfq pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "container \"dns-node-resolver\" in pod \"node-resolver-n4xfq\" is waiting to start: ContainerCreating" I0424 02:56:57.673055 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns-node-resolver\" in pod \"node-resolver-n4xfq\" is waiting to start: ContainerCreating" I0424 02:56:57.673066 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-zccfb pod in namespace openshift-dns (previous: false). I0424 02:56:57.845525 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:57.845542 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-zpz7f pod in namespace openshift-dns (previous: false). I0424 02:56:58.046847 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:58.046865 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6b6df6cddf-dd7m7 pod in namespace openshift-image-registry (previous: false). I0424 02:56:58.055254 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0424 02:56:58.239621 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:58.239639 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7bb69fc5c4-9pfrw pod in namespace openshift-image-registry (previous: false). I0424 02:56:58.454059 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-7bb69fc5c4-9pfrw pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-7bb69fc5c4-9pfrw\" is waiting to start: ContainerCreating" I0424 02:56:58.454081 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-7bb69fc5c4-9pfrw\" is waiting to start: ContainerCreating" I0424 02:56:58.454092 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7bb69fc5c4-rrnf8 pod in namespace openshift-image-registry (previous: false). I0424 02:56:58.461453 1 tasks_processing.go:74] worker 59 stopped. I0424 02:56:58.461497 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=b5b82290ea59f3bd35d5d88b8fc6ccfb9bccbd2ec96c920696f49c547df13a78 I0424 02:56:58.461540 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=325cc1f44ad08389600f091f34ed066fa0d87f483dac3c9ce8c276083bf722d5 I0424 02:56:58.461568 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0424 02:56:58.461593 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=4c2865dfbd8b62c0c0f6f2cc9d38bac6233c7ddb3cdf989faafc3a5827eb8b5c I0424 02:56:58.461609 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0424 02:56:58.461630 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=a4425732ce94f525681c55f0b6792725165ff35c27d90e0de833f2dd4caf79df I0424 02:56:58.461677 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=82260dfd883ed90e4abd9d1509d15f6c7b5be78d3a512a0076d665cc276c7574 I0424 02:56:58.461700 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=1cedad4a0788fd10ab3eb17886e5d74ea67ebd4d4a8a06badf474034719b83fb I0424 02:56:58.461714 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=b1b2d3926c44675be764e7b538bf8048e622393b1f17ba9d198e84e6854d3faa I0424 02:56:58.461731 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=833a4ea7a448ba2253f2311d7b2f76b6fbdacdca50adfc46f5b774d86f786cc4 I0424 02:56:58.461740 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0424 02:56:58.461756 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=40f4e30dfbe168d22c8b11d90059dee8dd3d186d76a8a0e4f675aa7980bb262d I0424 02:56:58.461765 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0424 02:56:58.461781 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=2a5616401286f7f23773a733651ff54d20e2835b8706ca50bfe05da8b4a81be5 I0424 02:56:58.461788 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0424 02:56:58.461800 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=1ab7ae925896f31a601b804ad573f5572ae6f39d0cf3a32bbc6115ca0c803c38 I0424 02:56:58.461808 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0424 02:56:58.461822 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=5c3036a26de0c8c7e02812a0c49d92a2ba1e86303349236b5187a0bec4565aa2 I0424 02:56:58.461968 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=552611347d31f97790608d8f8c4788f67d6a7994579303b63a283e67e0afcb5b I0424 02:56:58.461979 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0424 02:56:58.461987 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0424 02:56:58.462017 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0424 02:56:58.462040 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=6179d9c11b598bba7c5abc27e1c120adc93df4e849b37abc239e9c890d83d48b I0424 02:56:58.462063 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=3cf4f03d6797c0c139a83152da03155e7de0d7f972de58cb58f3bea8311c1812 I0424 02:56:58.462072 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0424 02:56:58.462093 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=44518f4a6dab6043b220c021d69360fe1bb823850562e0ca8454b8e25dad73aa I0424 02:56:58.462103 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0424 02:56:58.462115 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=9f5b6b2ccecd53554e9deb5223c3cde6a1b55be6e405dac7a9a2df6aaf351d6f I0424 02:56:58.462127 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=9b21015f8383548e998c5f7e931f5bfc102a9fec953a0ded3c495bfb8d1111db I0424 02:56:58.462143 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=4e34d7c2d43efc761e09128f6508ef0e5b840dd94a67446468579606552c3f39 I0424 02:56:58.462155 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=d1a22b246c80f55691d17834d928af52ddbca9455f526cc5354402bf678743a4 I0424 02:56:58.462168 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=6115460c04ed38bb041b222e8aeb11622fca3b5082364a46caf551f4f0fa63d0 I0424 02:56:58.462191 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=a19135dc90d6c8332281dce4f0638173632964aba16e706ed1862cba0fdc908a I0424 02:56:58.462212 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0424 02:56:58.462220 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0424 02:56:58.462227 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.864021726s to process 35 records I0424 02:56:58.639941 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:58.639957 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-2pm57 pod in namespace openshift-image-registry (previous: false). W0424 02:56:58.641395 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:56:58.847101 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-ca-2pm57 pod in namespace openshift-image-registry for failing operator node-ca (previous: false): "container \"node-ca\" in pod \"node-ca-2pm57\" is waiting to start: ContainerCreating" I0424 02:56:58.847117 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"node-ca\" in pod \"node-ca-2pm57\" is waiting to start: ContainerCreating" I0424 02:56:58.847126 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-wsjzs pod in namespace openshift-image-registry (previous: false). I0424 02:56:59.052228 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:59.052247 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-xsmjm pod in namespace openshift-image-registry (previous: false). I0424 02:56:59.244884 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:59.244901 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7bf489c4d9-bkqtb pod in namespace openshift-ingress (previous: false). I0424 02:56:59.444679 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7bf489c4d9-bkqtb pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7bf489c4d9-bkqtb\" is waiting to start: ContainerCreating" I0424 02:56:59.444704 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7bf489c4d9-bkqtb\" is waiting to start: ContainerCreating" I0424 02:56:59.444717 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8648f5cc7c-n6qpc pod in namespace openshift-ingress (previous: false). W0424 02:56:59.641133 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:56:59.644245 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0424 02:56:59.644258 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8648f5cc7c-q2j64 pod in namespace openshift-ingress (previous: false). I0424 02:56:59.845423 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-8648f5cc7c-q2j64 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-8648f5cc7c-q2j64\" is waiting to start: ContainerCreating" I0424 02:56:59.845440 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-8648f5cc7c-q2j64\" is waiting to start: ContainerCreating" I0424 02:56:59.845450 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-l2xc7 pod in namespace openshift-ingress-canary (previous: false). I0424 02:57:00.045710 1 tasks_processing.go:74] worker 5 stopped. I0424 02:57:00.045804 1 recorder.go:75] Recording events/openshift-dns with fingerprint=0754f78dbcfe816a318bd5965b72553804312085560fd03639ab42151ca39b18 I0424 02:57:00.045894 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=9c896274acc4e39bb9f64d05a5354eed6e95f3ae5461e5c33e33c26e3412bb8c I0424 02:57:00.045923 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=147d44b6bb8880e3b8f9e1db5630805bd3a9e04bec6fe46cd708c7182cbfc1c5 I0424 02:57:00.045979 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=10f1042f3a5a7dbc76b37d33650183f93537c7dfe6d3b2249c12bec57789f406 I0424 02:57:00.045999 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=d999da456967b5837fcaba268a0889cbcd4536e06036de1610ef9b335f500489 I0424 02:57:00.046006 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-l2xc7/serve-healthcheck-canary_current.log with fingerprint=726d854d41a32c157f6a939f1d2f5dc57a401c8a3a830b94179612107e8b35fe I0424 02:57:00.046013 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.448423782s to process 6 records W0424 02:57:00.640442 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 02:57:01.640564 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 02:57:01.640606 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 02:57:01.640624 1 tasks_processing.go:74] worker 50 stopped. E0424 02:57:01.640638 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 02:57:01.640657 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0424 02:57:01.640678 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0424 02:57:01.640696 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.040899267s to process 1 records I0424 02:57:09.046944 1 tasks_processing.go:74] worker 40 stopped. I0424 02:57:09.046983 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0424 02:57:09.046996 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.449226841s to process 1 records I0424 02:57:09.603831 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 02:57:09.806873 1 tasks_processing.go:74] worker 7 stopped. I0424 02:57:09.807180 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=698b3ed6ac2dcbbe305b75e3c37787498192c269ee2964c1d49d551d6371b6d6 I0424 02:57:09.807197 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.208699039s to process 1 records E0424 02:57:09.807251 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.21s with: function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0424 02:57:09.808362 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0424 02:57:09.808378 1 periodic.go:216] Running workloads gatherer I0424 02:57:09.808394 1 tasks_processing.go:45] number of workers: 2 I0424 02:57:09.808402 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:57:09.808406 1 tasks_processing.go:71] worker 1 working on workload_info task. I0424 02:57:09.808410 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:57:09.808419 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0424 02:57:09.840088 1 tasks_processing.go:74] worker 0 stopped. I0424 02:57:09.840103 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 31.655484ms to process 0 records I0424 02:57:09.892963 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0424 02:57:09.903002 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (11ms) I0424 02:57:09.913088 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (10ms) I0424 02:57:09.924462 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (11ms) I0424 02:57:09.934102 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (10ms) I0424 02:57:09.946883 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (13ms) I0424 02:57:09.956149 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (9ms) I0424 02:57:09.965642 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (9ms) I0424 02:57:09.975271 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (10ms) I0424 02:57:09.984593 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (9ms) I0424 02:57:09.993869 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (9ms) I0424 02:57:10.003584 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (10ms) I0424 02:57:10.025207 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 02:57:10.102749 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (99ms) I0424 02:57:10.203644 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (101ms) I0424 02:57:10.226608 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 02:57:10.304655 1 gather_workloads_info.go:387] No image sha256:ce138e8cf5b96557b1864ef6f27c2608bdca59be2611804366cef7169c36291e (101ms) I0424 02:57:10.405201 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (101ms) I0424 02:57:10.508756 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (104ms) I0424 02:57:10.603350 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (95ms) I0424 02:57:10.703370 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0424 02:57:10.803084 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (100ms) I0424 02:57:10.904338 1 gather_workloads_info.go:387] No image sha256:4b748615a43be416d52c33df03e2cdec89ededded1d6a2ade167c5a955d56e5f (101ms) I0424 02:57:11.004821 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (100ms) I0424 02:57:11.103110 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (98ms) I0424 02:57:11.203534 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (100ms) I0424 02:57:11.303374 1 gather_workloads_info.go:387] No image sha256:4733236617781e3469ffffb15e4daaa1f14ea8e1c52b426a3787a4f1f2945424 (100ms) I0424 02:57:11.403246 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (100ms) I0424 02:57:11.503066 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0424 02:57:11.604807 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (102ms) I0424 02:57:11.703335 1 gather_workloads_info.go:387] No image sha256:90a8ffd9643ebb16a6a8c04bb38cf9ed58903e9d3bf836c68f399193db5edaf6 (99ms) I0424 02:57:11.802652 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (99ms) I0424 02:57:11.902892 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (100ms) I0424 02:57:12.003295 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (100ms) I0424 02:57:12.103082 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (100ms) I0424 02:57:12.203692 1 gather_workloads_info.go:387] No image sha256:d9ed66918db4dfba8bd354c9ede4d676449fa0eccef649b3d8945cc1da1c60e3 (101ms) I0424 02:57:12.302689 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (99ms) I0424 02:57:12.406167 1 gather_workloads_info.go:387] No image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (103ms) I0424 02:57:12.406197 1 tasks_processing.go:74] worker 1 stopped. E0424 02:57:12.406209 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0424 02:57:12.406467 1 recorder.go:75] Recording config/workload_info with fingerprint=fe5b2841ae0e1ffeb41e30bfb8d7fa685fcef02f986a0d1ae2fedb40341f5881 I0424 02:57:12.406483 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.597783704s to process 1 records E0424 02:57:12.406507 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.598s with: function \"workload_info\" failed with an error" I0424 02:57:12.407615 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0424 02:57:12.407627 1 periodic.go:216] Running conditional gatherer I0424 02:57:12.413997 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0424 02:57:12.420738 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.8:42502->172.30.0.10:53: read: connection refused E0424 02:57:12.420988 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0424 02:57:12.421043 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0424 02:57:12.427258 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0424 02:57:12.427270 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427275 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427278 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427282 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427285 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427288 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427291 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427294 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0424 02:57:12.427297 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0424 02:57:12.427311 1 tasks_processing.go:45] number of workers: 3 I0424 02:57:12.427322 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 02:57:12.427326 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0424 02:57:12.427332 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:57:12.427342 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0424 02:57:12.427348 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0424 02:57:12.427348 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:57:12.427357 1 tasks_processing.go:74] worker 1 stopped. I0424 02:57:12.427397 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0424 02:57:12.427411 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.612µs to process 1 records I0424 02:57:12.427469 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0424 02:57:12.427477 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 603ns to process 1 records I0424 02:57:12.427484 1 tasks_processing.go:74] worker 0 stopped. I0424 02:57:12.427560 1 tasks_processing.go:74] worker 2 stopped. I0424 02:57:12.427572 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 228.357µs to process 0 records I0424 02:57:12.427589 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.8:42502->172.30.0.10:53: read: connection refused I0424 02:57:12.427607 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0424 02:57:12.451027 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=6ac8746d08d4b7a0cade4387261639d33bb290bef95c7ee2eb458a2c16c55bba I0424 02:57:12.451140 1 diskrecorder.go:70] Writing 99 records to /var/lib/insights-operator/insights-2026-04-24-025712.tar.gz I0424 02:57:12.457070 1 diskrecorder.go:51] Wrote 99 records to disk in 6ms I0424 02:57:12.457095 1 periodic.go:285] Gathering cluster info every 2h0m0s I0424 02:57:12.457111 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0424 02:57:25.400328 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 02:58:26.005420 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="ab81c73b4e88bd4aeebb8719c74784e59b57df4c698a5b43c63d4a0824123673") W0424 02:58:26.005453 1 builder.go:160] Restart triggered because of file /var/run/configmaps/service-ca-bundle/service-ca.crt was created I0424 02:58:26.005502 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="0b8e8a8c682f922368e6eb5fb556bc85b1910db8d75ea9317618fb268e9b8978")