W0702 02:13:03.768996 1 cmd.go:257] Using insecure, self-signed certificates I0702 02:13:04.960923 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:04.961285 1 observer_polling.go:159] Starting file observer I0702 02:13:05.609948 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0702 02:13:05.610237 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0702 02:13:05.611558 1 secure_serving.go:57] Forcing use of http/1.1 only W0702 02:13:05.611681 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0702 02:13:05.611715 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0702 02:13:05.611744 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0702 02:13:05.611777 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0702 02:13:05.611801 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0702 02:13:05.611824 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0702 02:13:05.611904 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0702 02:13:05.616296 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0702 02:13:05.616365 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"fc1ad981-3e5b-44ba-8e47-26b5d68aa868", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0702 02:13:05.617761 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0702 02:13:05.617773 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0702 02:13:05.617792 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 02:13:05.617780 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0702 02:13:05.617774 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0702 02:13:05.617826 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 02:13:05.618052 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-199280774/tls.crt::/tmp/serving-cert-199280774/tls.key" I0702 02:13:05.618292 1 secure_serving.go:213] Serving securely on [::]:8443 I0702 02:13:05.618326 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0702 02:13:05.625647 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0702 02:13:05.625671 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 02:13:05.625785 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0702 02:13:05.632908 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:13:05.632929 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 02:13:05.645108 1 secretconfigobserver.go:119] support secret does not exist I0702 02:13:05.662055 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:13:05.670324 1 secretconfigobserver.go:119] support secret does not exist I0702 02:13:05.673025 1 recorder.go:161] Pruning old reports every 6h2m5s, max age is 288h0m0s I0702 02:13:05.681212 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0702 02:13:05.681229 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0702 02:13:05.681241 1 periodic.go:209] Running clusterconfig gatherer I0702 02:13:05.681246 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0702 02:13:05.681258 1 insightsreport.go:296] Starting report retriever I0702 02:13:05.681269 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0702 02:13:05.681285 1 tasks_processing.go:45] number of workers: 32 I0702 02:13:05.681346 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 02:13:05.681514 1 tasks_processing.go:69] worker 3 listening for tasks. I0702 02:13:05.681568 1 tasks_processing.go:71] worker 3 working on certificate_signing_requests task. I0702 02:13:05.681611 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:13:05.681620 1 tasks_processing.go:69] worker 4 listening for tasks. I0702 02:13:05.681691 1 tasks_processing.go:71] worker 2 working on openstack_version task. I0702 02:13:05.681752 1 tasks_processing.go:69] worker 8 listening for tasks. I0702 02:13:05.681792 1 tasks_processing.go:69] worker 13 listening for tasks. I0702 02:13:05.681654 1 tasks_processing.go:69] worker 19 listening for tasks. I0702 02:13:05.681655 1 tasks_processing.go:69] worker 25 listening for tasks. I0702 02:13:05.681641 1 tasks_processing.go:69] worker 6 listening for tasks. I0702 02:13:05.681802 1 tasks_processing.go:69] worker 9 listening for tasks. I0702 02:13:05.681811 1 tasks_processing.go:69] worker 10 listening for tasks. I0702 02:13:05.682041 1 tasks_processing.go:69] worker 14 listening for tasks. I0702 02:13:05.682136 1 tasks_processing.go:69] worker 15 listening for tasks. I0702 02:13:05.682193 1 tasks_processing.go:69] worker 7 listening for tasks. I0702 02:13:05.681544 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:13:05.682231 1 tasks_processing.go:69] worker 20 listening for tasks. I0702 02:13:05.682243 1 tasks_processing.go:69] worker 16 listening for tasks. I0702 02:13:05.682309 1 tasks_processing.go:69] worker 17 listening for tasks. I0702 02:13:05.682337 1 tasks_processing.go:69] worker 18 listening for tasks. I0702 02:13:05.682375 1 tasks_processing.go:71] worker 1 working on silenced_alerts task. I0702 02:13:05.682435 1 tasks_processing.go:69] worker 22 listening for tasks. I0702 02:13:05.682449 1 tasks_processing.go:71] worker 22 working on proxies task. I0702 02:13:05.682453 1 tasks_processing.go:69] worker 29 listening for tasks. I0702 02:13:05.682474 1 tasks_processing.go:69] worker 12 listening for tasks. I0702 02:13:05.682383 1 tasks_processing.go:71] worker 18 working on monitoring_persistent_volumes task. I0702 02:13:05.682544 1 tasks_processing.go:71] worker 9 working on machines task. I0702 02:13:05.682581 1 tasks_processing.go:69] worker 27 listening for tasks. I0702 02:13:05.682638 1 tasks_processing.go:69] worker 28 listening for tasks. I0702 02:13:05.682666 1 tasks_processing.go:71] worker 7 working on openstack_controlplanes task. I0702 02:13:05.682777 1 tasks_processing.go:69] worker 23 listening for tasks. I0702 02:13:05.682810 1 tasks_processing.go:71] worker 14 working on node_logs task. I0702 02:13:05.683019 1 tasks_processing.go:69] worker 24 listening for tasks. I0702 02:13:05.683041 1 tasks_processing.go:71] worker 24 working on tsdb_status task. W0702 02:13:05.683086 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:05.683116 1 tasks_processing.go:71] worker 24 working on operators task. I0702 02:13:05.683126 1 tasks_processing.go:69] worker 11 listening for tasks. I0702 02:13:05.683149 1 tasks_processing.go:71] worker 27 working on networks task. I0702 02:13:05.683464 1 tasks_processing.go:71] worker 28 working on storage_cluster task. I0702 02:13:05.683517 1 tasks_processing.go:71] worker 29 working on image_registries task. I0702 02:13:05.683778 1 tasks_processing.go:71] worker 23 working on operators_pods_and_events task. I0702 02:13:05.683949 1 tasks_processing.go:71] worker 0 working on openshift_machine_api_events task. I0702 02:13:05.684147 1 tasks_processing.go:71] worker 15 working on sap_config task. I0702 02:13:05.682492 1 tasks_processing.go:69] worker 30 listening for tasks. I0702 02:13:05.682369 1 tasks_processing.go:71] worker 17 working on machine_sets task. I0702 02:13:05.684363 1 tasks_processing.go:71] worker 20 working on overlapping_namespace_uids task. I0702 02:13:05.682560 1 tasks_processing.go:69] worker 31 listening for tasks. I0702 02:13:05.682342 1 tasks_processing.go:69] worker 21 listening for tasks. I0702 02:13:05.682500 1 tasks_processing.go:69] worker 26 listening for tasks. W0702 02:13:05.682498 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:05.682509 1 tasks_processing.go:71] worker 4 working on mutating_webhook_configurations task. I0702 02:13:05.682514 1 tasks_processing.go:71] worker 8 working on machine_configs task. I0702 02:13:05.682518 1 tasks_processing.go:71] worker 13 working on authentication task. I0702 02:13:05.682526 1 tasks_processing.go:71] worker 19 working on infrastructures task. I0702 02:13:05.682529 1 tasks_processing.go:71] worker 25 working on support_secret task. I0702 02:13:05.682538 1 tasks_processing.go:71] worker 6 working on ingress_certificates task. I0702 02:13:05.681644 1 tasks_processing.go:69] worker 5 listening for tasks. I0702 02:13:05.684427 1 tasks_processing.go:71] worker 16 working on storage_classes task. I0702 02:13:05.684545 1 tasks_processing.go:71] worker 5 working on image_pruners task. I0702 02:13:05.684442 1 tasks_processing.go:71] worker 10 working on nodenetworkconfigurationpolicies task. I0702 02:13:05.684468 1 tasks_processing.go:71] worker 12 working on sap_pods task. I0702 02:13:05.684473 1 tasks_processing.go:71] worker 30 working on clusterroles task. I0702 02:13:05.684480 1 tasks_processing.go:71] worker 11 working on install_plans task. I0702 02:13:05.684483 1 tasks_processing.go:71] worker 31 working on nodenetworkstates task. I0702 02:13:05.684490 1 tasks_processing.go:71] worker 26 working on active_alerts task. W0702 02:13:05.684839 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:05.684495 1 tasks_processing.go:71] worker 21 working on service_accounts task. I0702 02:13:05.685105 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 53.005µs to process 0 records I0702 02:13:05.685121 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 2.053893ms to process 0 records I0702 02:13:05.685127 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 22.077µs to process 0 records I0702 02:13:05.685135 1 tasks_processing.go:71] worker 26 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0702 02:13:05.685372 1 tasks_processing.go:71] worker 1 working on ingress task. I0702 02:13:05.685989 1 tasks_processing.go:71] worker 2 working on dvo_metrics task. I0702 02:13:05.686003 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 4.232704ms to process 0 records I0702 02:13:05.690662 1 tasks_processing.go:71] worker 28 working on aggregated_monitoring_cr_names task. I0702 02:13:05.690715 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 7.145215ms to process 0 records I0702 02:13:05.690732 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 8.037639ms to process 0 records I0702 02:13:05.690782 1 tasks_processing.go:71] worker 7 working on container_images task. I0702 02:13:05.690811 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0702 02:13:05.690820 1 tasks_processing.go:71] worker 22 working on qemu_kubevirt_launcher_logs task. I0702 02:13:05.690830 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 02:13:05.690836 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 02:13:05.690841 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 02:13:05.690860 1 controller.go:489] The operator is still being initialized I0702 02:13:05.690869 1 controller.go:512] The operator is healthy I0702 02:13:05.690930 1 recorder.go:75] Recording config/proxy with fingerprint=77660b69ef06de7eaa0cfcb89b1627e2ff29b6ebce8226f578f434b7e77f7bad I0702 02:13:05.690953 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 8.361802ms to process 1 records E0702 02:13:05.690970 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0702 02:13:05.690981 1 gather.go:177] gatherer "clusterconfig" function "machines" took 8.259184ms to process 0 records I0702 02:13:05.690998 1 tasks_processing.go:71] worker 9 working on ceph_cluster task. I0702 02:13:05.697520 1 tasks_processing.go:71] worker 27 working on olm_operators task. I0702 02:13:05.697785 1 recorder.go:75] Recording config/network with fingerprint=13cd34e141d84bd5739bd4dd399c9fe1c79cdfdc2ee256f2113a4f24dce3b3e4 I0702 02:13:05.697802 1 gather.go:177] gatherer "clusterconfig" function "networks" took 14.350203ms to process 1 records I0702 02:13:05.703762 1 tasks_processing.go:71] worker 17 working on openshift_logging task. I0702 02:13:05.703780 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 19.452463ms to process 0 records I0702 02:13:05.703796 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 19.562708ms to process 0 records I0702 02:13:05.703807 1 tasks_processing.go:71] worker 15 working on config_maps task. I0702 02:13:05.708547 1 tasks_processing.go:71] worker 13 working on machine_healthchecks task. I0702 02:13:05.708845 1 recorder.go:75] Recording config/authentication with fingerprint=b76a58a97efab0f02342c14bbd4261db84793e1a841d88e113d99303906d937b I0702 02:13:05.708866 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 24.026456ms to process 1 records I0702 02:13:05.712238 1 tasks_processing.go:71] worker 12 working on openstack_dataplanenodesets task. I0702 02:13:05.712243 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 27.563074ms to process 0 records I0702 02:13:05.717624 1 tasks_processing.go:71] worker 17 working on pdbs task. I0702 02:13:05.717627 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 13.847178ms to process 0 records I0702 02:13:05.717864 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 02:13:05.717880 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 02:13:05.717919 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0702 02:13:05.718001 1 tasks_processing.go:71] worker 18 working on metrics task. I0702 02:13:05.718066 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 35.450994ms to process 0 records W0702 02:13:05.718087 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:05.718101 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 33.294462ms to process 0 records I0702 02:13:05.718165 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 108.27µs to process 0 records I0702 02:13:05.718174 1 tasks_processing.go:71] worker 18 working on schedulers task. I0702 02:13:05.718107 1 tasks_processing.go:71] worker 31 working on feature_gates task. I0702 02:13:05.718282 1 tasks_processing.go:71] worker 9 working on validating_webhook_configurations task. I0702 02:13:05.718405 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 27.275525ms to process 0 records I0702 02:13:05.718547 1 tasks_processing.go:71] worker 10 working on oauths task. I0702 02:13:05.718560 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 33.923992ms to process 0 records I0702 02:13:05.718676 1 tasks_processing.go:71] worker 20 working on container_runtime_configs task. I0702 02:13:05.718973 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0702 02:13:05.719039 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 34.187368ms to process 1 records I0702 02:13:05.720863 1 tasks_processing.go:71] worker 25 working on nodes task. E0702 02:13:05.720893 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0702 02:13:05.720968 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 36.33456ms to process 0 records I0702 02:13:05.722530 1 tasks_processing.go:71] worker 19 working on machine_config_pools task. I0702 02:13:05.723216 1 recorder.go:75] Recording config/infrastructure with fingerprint=362bf7b860d99d9ef51b509b617e62922e4aa79df800925c73f1791a8b77f7c1 I0702 02:13:05.723232 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 38.025154ms to process 1 records I0702 02:13:05.723316 1 tasks_processing.go:71] worker 1 working on lokistack task. I0702 02:13:05.723366 1 recorder.go:75] Recording config/ingress with fingerprint=53bc3541ca1b97ceb32ee0b147ca2d5bb85b2c0931a6980ef2d096f16044776c I0702 02:13:05.723380 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 37.237102ms to process 1 records I0702 02:13:05.723458 1 tasks_processing.go:71] worker 5 working on crds task. I0702 02:13:05.723566 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c0e624c78dce58ad9ed0df78325e09e964cad7c096640d7626010eaa43a2bf7e I0702 02:13:05.723579 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 38.396075ms to process 1 records I0702 02:13:05.723937 1 tasks_processing.go:71] worker 3 working on sap_datahubs task. I0702 02:13:05.723952 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 42.311595ms to process 0 records I0702 02:13:05.726427 1 base_controller.go:82] Caches are synced for ConfigController I0702 02:13:05.726443 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0702 02:13:05.728825 1 tasks_processing.go:71] worker 13 working on cost_management_metrics_configs task. E0702 02:13:05.728834 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0702 02:13:05.729016 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 20.259799ms to process 0 records I0702 02:13:05.729037 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 16.736247ms to process 0 records I0702 02:13:05.729044 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 10.305948ms to process 0 records I0702 02:13:05.729053 1 tasks_processing.go:71] worker 20 working on jaegers task. I0702 02:13:05.729166 1 tasks_processing.go:71] worker 12 working on openstack_dataplanedeployments task. I0702 02:13:05.729376 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0702 02:13:05.729464 1 operator.go:288] started I0702 02:13:05.729477 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0702 02:13:05.729490 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0702 02:13:05.736648 1 tasks_processing.go:71] worker 17 working on version task. I0702 02:13:05.736771 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=5e63ea5a83644da0859891819adc8cb3ab9e154e747f168574b6691a4bff48e6 I0702 02:13:05.736810 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=0f5ff6c655f6f8d3c533af61142fdbe1bbfe867f42e18d3637721ae739f3fb25 I0702 02:13:05.736841 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=db35a428fb276671d8212c3f8c53f058a5dbd4bcad18991196b897ddda0bb80d I0702 02:13:05.736855 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 19.005142ms to process 3 records I0702 02:13:05.736867 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 52.736643ms to process 0 records I0702 02:13:05.736876 1 tasks_processing.go:71] worker 0 working on pod_network_connectivity_checks task. I0702 02:13:05.737101 1 tasks_processing.go:71] worker 18 working on image task. I0702 02:13:05.737248 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=8a37ee77a19bb3f1dc01181130186523ff15ba9ffac89360632626f3be47e007 I0702 02:13:05.737314 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 18.919907ms to process 1 records I0702 02:13:05.737326 1 gather_logs.go:145] no pods in namespace were found I0702 02:13:05.737338 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 54.293112ms to process 0 records I0702 02:13:05.737345 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 46.507285ms to process 0 records I0702 02:13:05.737351 1 tasks_processing.go:71] worker 14 working on cluster_apiserver task. I0702 02:13:05.737443 1 tasks_processing.go:71] worker 22 working on machine_autoscalers task. I0702 02:13:05.737487 1 tasks_processing.go:74] worker 29 stopped. I0702 02:13:05.737877 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=3f575e978ce90adb7a98a9f283b24b4e833b443d3634ff101f93e30c140a2829 I0702 02:13:05.737894 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 53.948091ms to process 1 records I0702 02:13:05.743297 1 tasks_processing.go:74] worker 4 stopped. I0702 02:13:05.743965 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=6efd6a028fdd92efc809c1a6384b8c8efb334490760e1e9143d0c023813c7280 I0702 02:13:05.744075 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=c8295eaedf8bd81e6ef169eafdcbf7d6bc46d8a80899f1c4ee435e8e94222cae I0702 02:13:05.744177 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=e289c6e5c3c97a0eb9f3fc0c29ffd0384f84ebe4ff5c7e0ff311794b388836bb I0702 02:13:05.744232 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 58.802125ms to process 3 records I0702 02:13:05.744262 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 45.789626ms to process 0 records I0702 02:13:05.744468 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=bffd27bf1598b0fc0d1a73cb507981a1e51e6f4c6dacad72f9e7ce5843a1d81c I0702 02:13:05.744512 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=79008a39a0b1c5fb87e68f7a4c6b5c38fdc6d815bea89686be1b2b2df054faf1 I0702 02:13:05.744554 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 59.62289ms to process 2 records I0702 02:13:05.744619 1 tasks_processing.go:74] worker 16 stopped. I0702 02:13:05.744654 1 tasks_processing.go:74] worker 27 stopped. I0702 02:13:05.745290 1 tasks_processing.go:74] worker 12 stopped. I0702 02:13:05.745347 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 16.094839ms to process 0 records I0702 02:13:05.745651 1 tasks_processing.go:74] worker 9 stopped. I0702 02:13:05.746233 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=72ed4cb9ed113f7da74acb3cf5f4b4116653a08ada468b40450bf2433a275727 I0702 02:13:05.746353 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=80ce837067c1cfc2383744438efd2a43fc8f10f11cf1fbea1385608bac24c19b I0702 02:13:05.746388 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=6d73209bf1c107faf3516f0e6a715b6963a63c15473053554e0e8c54bc1f2a16 I0702 02:13:05.746426 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=ab5b53dfb563d781e55954071a167c1456b182e1bd2dbafdc22240b102c7da5b I0702 02:13:05.746460 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=13db0eaaad84fb771b093abbb4695fc2a3e24ed576d0f3a4b60ced98bfc771ab I0702 02:13:05.746494 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=6c3a2df865d79366569bd44ff99eea0a2e9c801bbe5b88c4229115aa4d8af751 I0702 02:13:05.746527 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=48842de30a060158341af99d06fa8f408cc5def7a3e5872a7b1a97ba82b4f1f0 I0702 02:13:05.746575 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=f188b693159514ebfe3a93038ae5eb02175314ccb3aa86851ba5bbad1fd75bab I0702 02:13:05.746634 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=dad98edb284b98b05ae620c5b40b6fc4c1e3b6969253de0d3a035cc5bd839257 I0702 02:13:05.746673 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=613eba69aee32e278dc759bd6a9b33150f4e073b434df5054b8b66500a4aacc3 I0702 02:13:05.746731 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=bddcc3b5fbd0670ad2c10b9cdcca710f84bc262fb0f34c2dbc11d21cc2ee9a13 I0702 02:13:05.746751 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 27.344179ms to process 11 records I0702 02:13:05.746829 1 tasks_processing.go:74] worker 31 stopped. I0702 02:13:05.746866 1 recorder.go:75] Recording config/featuregate with fingerprint=bc4451ad638852bee6cd16b5d6b5d9925ef2e590cb0d685fe04967dee00c6953 I0702 02:13:05.746883 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 27.980738ms to process 1 records I0702 02:13:05.748552 1 tasks_processing.go:74] worker 1 stopped. I0702 02:13:05.748575 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 25.220755ms to process 0 records I0702 02:13:05.749005 1 tasks_processing.go:74] worker 10 stopped. I0702 02:13:05.749338 1 recorder.go:75] Recording config/oauth with fingerprint=c13f22cdde12c5ba78ea5201fa1fabe98f96f7daa2ec44343ede86d3431a72c1 I0702 02:13:05.749359 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 30.417588ms to process 1 records I0702 02:13:05.749411 1 tasks_processing.go:74] worker 3 stopped. I0702 02:13:05.749425 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 25.460514ms to process 0 records W0702 02:13:05.750246 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:13:05.752441 1 tasks_processing.go:74] worker 13 stopped. I0702 02:13:05.752457 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.549347ms to process 0 records I0702 02:13:05.752468 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 23.377285ms to process 0 records I0702 02:13:05.752475 1 tasks_processing.go:74] worker 20 stopped. I0702 02:13:05.752935 1 tasks_processing.go:74] worker 30 stopped. I0702 02:13:05.753219 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=edcdaa8be8fa683254d7b3bdb0f88a8d709ceb38d000643fd03d0267937332ea I0702 02:13:05.753375 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=17b985fb17c5103114417cc2c2e1b92579a7772fec198c32477bb2c0515e7475 I0702 02:13:05.753390 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 68.224567ms to process 2 records I0702 02:13:05.758445 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:13:05.758780 1 prometheus_rules.go:88] Prometheus rules successfully created I0702 02:13:05.770202 1 tasks_processing.go:74] worker 22 stopped. I0702 02:13:05.770301 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 32.742768ms to process 0 records I0702 02:13:05.770339 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 47.681024ms to process 0 records I0702 02:13:05.770352 1 tasks_processing.go:74] worker 19 stopped. I0702 02:13:05.770527 1 tasks_processing.go:74] worker 0 stopped. E0702 02:13:05.770556 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0702 02:13:05.770574 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 33.640321ms to process 0 records I0702 02:13:05.770707 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 02:13:05.770725 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 02:13:05.770732 1 controller.go:212] Source scaController *sca.Controller is not ready I0702 02:13:05.770739 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0702 02:13:05.770744 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 02:13:05.770773 1 controller.go:489] The operator is still being initialized I0702 02:13:05.770785 1 controller.go:512] The operator is healthy I0702 02:13:05.770982 1 tasks_processing.go:74] worker 28 stopped. I0702 02:13:05.771007 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 80.298232ms to process 0 records I0702 02:13:05.771651 1 tasks_processing.go:74] worker 26 stopped. I0702 02:13:05.771670 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 86.506137ms to process 0 records I0702 02:13:05.773612 1 tasks_processing.go:74] worker 7 stopped. I0702 02:13:05.775360 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-5thn5 with fingerprint=909053255cea44f809ff6e0ccce56cd67e33e6d8a8b205fc4053cd1e535f0d37 I0702 02:13:05.775417 1 recorder.go:75] Recording config/running_containers with fingerprint=b1e159f09814b59b11a75c2a70da23fc9e10c74bf16baf327f1620d40c3ce659 I0702 02:13:05.775427 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 82.789175ms to process 2 records E0702 02:13:05.778134 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27533dc089-4380-41ca-9d1c-a1323d76498b%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.130.0.11:47823->172.30.0.10:53: read: connection refused I0702 02:13:05.778151 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27533dc089-4380-41ca-9d1c-a1323d76498b%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.130.0.11:47823->172.30.0.10:53: read: connection refused I0702 02:13:05.780020 1 tasks_processing.go:74] worker 18 stopped. I0702 02:13:05.780122 1 recorder.go:75] Recording config/image with fingerprint=97f5378a31b0404f7feb418689fdca41a9dc2c52ab438bd1a25f4a2606b23077 I0702 02:13:05.780136 1 gather.go:177] gatherer "clusterconfig" function "image" took 42.896205ms to process 1 records I0702 02:13:05.780208 1 tasks_processing.go:74] worker 14 stopped. I0702 02:13:05.780386 1 recorder.go:75] Recording config/apiserver with fingerprint=dd2736a7f8fffcbd4259f27212dfd2727f2328f3717f64c55e3e244c8f2f6dcf I0702 02:13:05.780405 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 42.84327ms to process 1 records I0702 02:13:05.790714 1 tasks_processing.go:74] worker 5 stopped. I0702 02:13:05.791477 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=fe724d7d2a1020dc09e9e7ec38088691eefe857bf1d60af2d86d443f13a14cd7 I0702 02:13:05.791831 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=0d00648da7e93380041e1b76ee4b40587754a211404aa555eb7a11954f1de5ad I0702 02:13:05.791850 1 gather.go:177] gatherer "clusterconfig" function "crds" took 67.239555ms to process 2 records I0702 02:13:05.803518 1 tasks_processing.go:74] worker 17 stopped. I0702 02:13:05.803886 1 recorder.go:75] Recording config/version with fingerprint=b7d1529bc5029c5664dbb2388b5b14dad185718601cd81628f3002ed1214a9c2 I0702 02:13:05.803905 1 recorder.go:75] Recording config/id with fingerprint=ab78cbed52b0b56b3eb04d2475fad9e8caf0b89e3ff1fae59f6cc53ee8a791ee I0702 02:13:05.803912 1 gather.go:177] gatherer "clusterconfig" function "version" took 66.850817ms to process 2 records I0702 02:13:05.829697 1 base_controller.go:82] Caches are synced for LoggingSyncer I0702 02:13:05.829717 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0702 02:13:05.833175 1 tasks_processing.go:74] worker 25 stopped. I0702 02:13:05.833483 1 recorder.go:75] Recording config/node/ip-10-0-0-102.ec2.internal with fingerprint=040734bfc79f33f38eaa26aeae05069bc02ad31d95dff42c9069090b756b17e5 I0702 02:13:05.833540 1 recorder.go:75] Recording config/node/ip-10-0-1-31.ec2.internal with fingerprint=1430c93a411504b9b9afaac01c67111d3463dd03614796f8099d29e37dc66148 I0702 02:13:05.833611 1 recorder.go:75] Recording config/node/ip-10-0-2-199.ec2.internal with fingerprint=e73541e7a8804f4311ee347e4336f1ec74b674c4ea02ab0665c6c1f62ad5eb33 I0702 02:13:05.833621 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 112.248573ms to process 3 records I0702 02:13:05.844198 1 tasks_processing.go:74] worker 15 stopped. E0702 02:13:05.844215 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0702 02:13:05.844221 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0702 02:13:05.844224 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0702 02:13:05.844236 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0702 02:13:05.844275 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0702 02:13:05.844283 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0702 02:13:05.844287 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0702 02:13:05.844291 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0702 02:13:05.844331 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0702 02:13:05.844338 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0702 02:13:05.844343 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 140.373267ms to process 7 records I0702 02:13:05.860401 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0702 02:13:05.864229 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.130.0.11:39682->172.30.0.10:53: read: connection refused I0702 02:13:05.864244 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.130.0.11:39682->172.30.0.10:53: read: connection refused I0702 02:13:05.865969 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0702 02:13:05.874652 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0702 02:13:05.890221 1 tasks_processing.go:74] worker 6 stopped. E0702 02:13:05.890236 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0702 02:13:05.890243 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2r9ne0rg647slnpt68mtrpp6jrvnu4so-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2r9ne0rg647slnpt68mtrpp6jrvnu4so-primary-cert-bundle-secret" not found I0702 02:13:05.890307 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=5a51b1bd492340ea23d709dd1a67a8f2653530df1fdab2f117eeab42dd88e4ea I0702 02:13:05.890321 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 205.693798ms to process 1 records I0702 02:13:06.150243 1 gather_cluster_operator_pods_and_events.go:121] Found 18 pods with 21 containers I0702 02:13:06.150260 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1198372 bytes I0702 02:13:06.151034 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-l4ptc pod in namespace openshift-dns (previous: false). I0702 02:13:06.380996 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-l4ptc pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-l4ptc\" is waiting to start: ContainerCreating" I0702 02:13:06.381013 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-l4ptc\" is waiting to start: ContainerCreating" I0702 02:13:06.381025 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-l4ptc pod in namespace openshift-dns (previous: false). I0702 02:13:06.552789 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-l4ptc pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-l4ptc\" is waiting to start: ContainerCreating" I0702 02:13:06.552807 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-l4ptc\" is waiting to start: ContainerCreating" I0702 02:13:06.552816 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-pxnqg pod in namespace openshift-dns (previous: false). W0702 02:13:06.749445 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:13:06.770841 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pxnqg pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-pxnqg\" is waiting to start: ContainerCreating" I0702 02:13:06.770859 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-pxnqg\" is waiting to start: ContainerCreating" I0702 02:13:06.770866 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-pxnqg pod in namespace openshift-dns (previous: false). I0702 02:13:06.787138 1 tasks_processing.go:74] worker 8 stopped. I0702 02:13:06.787180 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0702 02:13:06.787200 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 1.102633611s to process 1 records I0702 02:13:06.958980 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pxnqg pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-pxnqg\" is waiting to start: ContainerCreating" I0702 02:13:06.959005 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-pxnqg\" is waiting to start: ContainerCreating" I0702 02:13:06.959015 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-qcgrf pod in namespace openshift-dns (previous: false). I0702 02:13:07.154889 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0702 02:13:07.174165 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-qcgrf pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-qcgrf\" is waiting to start: ContainerCreating" I0702 02:13:07.174181 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-qcgrf\" is waiting to start: ContainerCreating" I0702 02:13:07.174189 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-qcgrf pod in namespace openshift-dns (previous: false). I0702 02:13:07.352461 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-qcgrf pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-qcgrf\" is waiting to start: ContainerCreating" I0702 02:13:07.352476 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-qcgrf\" is waiting to start: ContainerCreating" I0702 02:13:07.352486 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-44gcw pod in namespace openshift-dns (previous: false). I0702 02:13:07.551888 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:07.551915 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-crbnk pod in namespace openshift-dns (previous: false). W0702 02:13:07.749798 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:13:07.752614 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:07.752628 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-l8rpb pod in namespace openshift-dns (previous: false). I0702 02:13:07.757183 1 tasks_processing.go:74] worker 24 stopped. I0702 02:13:07.757225 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=1471602e9ef3d394601c7bd408a386a6ad5c50e27e325fed0a1f2c90134de613 I0702 02:13:07.757254 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=c3f8d4a7c9808b40a57dd63e09cadb3795d32aa0fcb1508de7306466ab647755 I0702 02:13:07.757281 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0702 02:13:07.757310 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=737a609cb91644f7ef75084e045c321fe20f4c78b5a2c7a6c877be6cd40a415a I0702 02:13:07.757327 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0702 02:13:07.757366 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=5514f9612a931140759706630c665905d5bbfba800af579f5a1d905b6f7da6de I0702 02:13:07.757396 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=54d9202c8601445937bcb778ee4f657384cf291fb2834d5d832237ad720a02ec I0702 02:13:07.757421 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=1583a68b092eb2428b5b04c1ca0607797ccd8de693c0aca84d3544225f740774 I0702 02:13:07.757435 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=4674f2ad595616f107b40e446e078c241ee068981607d834089ac9b0fea33d9b I0702 02:13:07.757452 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=94d95b4b3a6895f4bc20a901d8deab13816fad633eea9af6334d6781e9f5697e I0702 02:13:07.757461 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0702 02:13:07.757487 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=05542185fa9ab498048426c5e935850ae8cb981913bb06698373eb1b0e209a0c I0702 02:13:07.757498 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0702 02:13:07.757513 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=bd8d5f8fe6f9b773d93b7d2f553277b2232ff8e6d010c176d6924220e9614cae I0702 02:13:07.757522 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0702 02:13:07.757540 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=3cd71d3746991c02e681dd8ed3ecdca7d5f3ca7d0e4f47b5f7ec78418731c510 I0702 02:13:07.757549 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0702 02:13:07.757564 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=75b0ac5e61bd6c3943a893a4f0b861f8a9444df1cd8d1583d0ae9fd72255dd77 I0702 02:13:07.757725 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=609c711e54d8cd9201e31f1e38e02e469cacd7455a719bdcca1c7be54d96c437 I0702 02:13:07.757738 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0702 02:13:07.757745 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0702 02:13:07.757766 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0702 02:13:07.757812 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=9713b99e76094cb6b64495311eedd371da0b29501174affaf20f241f94c468c8 I0702 02:13:07.757837 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=f19684871f67154b84254335a2aadc16c022f6520280ec5af18d03ab00a71c2e I0702 02:13:07.757853 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0702 02:13:07.757872 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=d58bed552c752768b07546281480f209d065b892068257967a76bd939a254c3b I0702 02:13:07.757882 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0702 02:13:07.757895 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=fd7eca6a3cafba6cc54ad8b832e3dcebc26de6e5a555f2a4e6ea1961794ccc5e I0702 02:13:07.757911 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=5e197e9fbcf2eeef7c487852d62f25337059104c83ca0389a02ca5b0f3a302a0 I0702 02:13:07.757926 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=9aafa8d8c8d732aee580c5aad49de930777bacf3c44df9bfed7515d6b4fba5df I0702 02:13:07.757943 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=8f6ad4151e83447c97ea95c3124e44b370d28b6bb6869c74ea22b549daeda693 I0702 02:13:07.757964 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=d4c1fc7e2c14687aafff9265eec4c4d91c2817a9cf10e3f6b940dbb373501f5c I0702 02:13:07.757974 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0702 02:13:07.757998 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=e4238ef66b84d99d8908430cfcd53c8da6240835d6fa7db88ad14f5f7d423f99 I0702 02:13:07.758015 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0702 02:13:07.758024 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0702 02:13:07.758031 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.074048508s to process 36 records I0702 02:13:07.952636 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:07.952693 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5b877cc7c6-ztskz pod in namespace openshift-image-registry (previous: false). I0702 02:13:08.150925 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-5b877cc7c6-ztskz pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-5b877cc7c6-ztskz\" is waiting to start: ContainerCreating" I0702 02:13:08.150943 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-5b877cc7c6-ztskz\" is waiting to start: ContainerCreating" I0702 02:13:08.151003 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6ddb857f9f-lqxh5 pod in namespace openshift-image-registry (previous: false). I0702 02:13:08.353617 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6ddb857f9f-lqxh5 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6ddb857f9f-lqxh5\" is waiting to start: ContainerCreating" I0702 02:13:08.353636 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6ddb857f9f-lqxh5\" is waiting to start: ContainerCreating" I0702 02:13:08.353683 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6ddb857f9f-nrtkd pod in namespace openshift-image-registry (previous: false). I0702 02:13:08.551887 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6ddb857f9f-nrtkd pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6ddb857f9f-nrtkd\" is waiting to start: ContainerCreating" I0702 02:13:08.551904 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6ddb857f9f-nrtkd\" is waiting to start: ContainerCreating" I0702 02:13:08.551913 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-5jw4p pod in namespace openshift-image-registry (previous: false). W0702 02:13:08.749152 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:13:08.756761 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:08.756792 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-qkjdg pod in namespace openshift-image-registry (previous: false). I0702 02:13:08.951889 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:08.951908 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-vr2nk pod in namespace openshift-image-registry (previous: false). I0702 02:13:09.151622 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 02:13:09.151644 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5ccf7b5f8f-fgl5k pod in namespace openshift-ingress (previous: false). I0702 02:13:09.354417 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5ccf7b5f8f-fgl5k pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5ccf7b5f8f-fgl5k\" is waiting to start: ContainerCreating" I0702 02:13:09.354433 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5ccf7b5f8f-fgl5k\" is waiting to start: ContainerCreating" I0702 02:13:09.354443 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-fdfd7fc66-58vns pod in namespace openshift-ingress (previous: false). I0702 02:13:09.556422 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-fdfd7fc66-58vns pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-fdfd7fc66-58vns\" is waiting to start: ContainerCreating" I0702 02:13:09.556441 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-fdfd7fc66-58vns\" is waiting to start: ContainerCreating" I0702 02:13:09.556452 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-fdfd7fc66-hqgs9 pod in namespace openshift-ingress (previous: false). W0702 02:13:09.749408 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:13:09.754946 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-fdfd7fc66-hqgs9 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-fdfd7fc66-hqgs9\" is waiting to start: ContainerCreating" I0702 02:13:09.754960 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-fdfd7fc66-hqgs9\" is waiting to start: ContainerCreating" I0702 02:13:09.754969 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-5jfrl pod in namespace openshift-ingress-canary (previous: false). I0702 02:13:09.952542 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-5jfrl pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5jfrl\" is waiting to start: ContainerCreating" I0702 02:13:09.952559 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5jfrl\" is waiting to start: ContainerCreating" I0702 02:13:09.952570 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-9ptvk pod in namespace openshift-ingress-canary (previous: false). I0702 02:13:10.151767 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-9ptvk pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-9ptvk\" is waiting to start: ContainerCreating" I0702 02:13:10.151788 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-9ptvk\" is waiting to start: ContainerCreating" I0702 02:13:10.151800 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-fclhp pod in namespace openshift-ingress-canary (previous: false). I0702 02:13:10.353603 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-fclhp pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-fclhp\" is waiting to start: ContainerCreating" I0702 02:13:10.353621 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-fclhp\" is waiting to start: ContainerCreating" I0702 02:13:10.353642 1 tasks_processing.go:74] worker 23 stopped. I0702 02:13:10.353754 1 recorder.go:75] Recording events/openshift-dns with fingerprint=838a4a571e1b170628b466bb2fed7a3f1bc4a32ce9fa6eeb87f0cdd460a9e4f7 I0702 02:13:10.353850 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=c2a630a51eaf80d942bf4daaea219bf9004ccfef76b953126aa1441eeffde69a I0702 02:13:10.353879 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=692316e8de0d6d17fe0a772d5b0732f0c95dddab2925d5df6b11d396a12580f6 I0702 02:13:10.353927 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=15da83cd80eacc4892cdcd91e91360cf4f4b38bd5fbcf7b7f096660ea9de1792 I0702 02:13:10.353944 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=f574b4a10701e9a665e4107bef7a05efc74ba78d1c16208074e03f0c1826b427 I0702 02:13:10.354105 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5b877cc7c6-ztskz with fingerprint=60d45f67fe22cbe227097a67b973e9b455751ba105f2a2882fc812622aeeeee4 I0702 02:13:10.354211 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-6ddb857f9f-lqxh5 with fingerprint=e9b1699d99f0eb0d6e5ba08be5ed91e3a0bd82c14c9ac941a72808a8e03270bd I0702 02:13:10.354300 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-6ddb857f9f-nrtkd with fingerprint=d8b8e91ea4fb7ce8c69fcf9a55a52d527f2b654bfb33e11baefb384bccfc224f I0702 02:13:10.354309 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.669809066s to process 8 records W0702 02:13:10.748565 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0702 02:13:10.748607 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 02:13:10.748625 1 tasks_processing.go:74] worker 2 stopped. E0702 02:13:10.748635 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 02:13:10.748647 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0702 02:13:10.748661 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0702 02:13:10.748673 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.062617979s to process 1 records I0702 02:13:18.247306 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:13:18.822846 1 tasks_processing.go:74] worker 11 stopped. I0702 02:13:18.822888 1 recorder.go:75] Recording config/installplans with fingerprint=e3c18441424aaa81f54ea9723d194eacb5824eb47178047bd893de11825bc8a2 I0702 02:13:18.822902 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.13811486s to process 1 records I0702 02:13:19.495359 1 tasks_processing.go:74] worker 21 stopped. I0702 02:13:19.495648 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=2795f4fb1eec9e9b515e9d465dc026684b73654f00c1120ebc66fca9dd8c1a2c I0702 02:13:19.495665 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.810494776s to process 1 records E0702 02:13:19.495718 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.814s with: function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0702 02:13:19.496817 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0702 02:13:19.496832 1 periodic.go:209] Running workloads gatherer I0702 02:13:19.496846 1 tasks_processing.go:45] number of workers: 2 I0702 02:13:19.496856 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:13:19.496860 1 tasks_processing.go:71] worker 1 working on workload_info task. I0702 02:13:19.496874 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:13:19.496894 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0702 02:13:19.527215 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0702 02:13:19.533690 1 tasks_processing.go:74] worker 0 stopped. I0702 02:13:19.533704 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 36.782236ms to process 0 records I0702 02:13:19.536684 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (10ms) I0702 02:13:19.545983 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0702 02:13:19.555062 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0702 02:13:19.564376 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (9ms) I0702 02:13:19.573927 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (10ms) I0702 02:13:19.583820 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (10ms) I0702 02:13:19.595251 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (11ms) I0702 02:13:19.609152 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (14ms) I0702 02:13:19.618289 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (9ms) I0702 02:13:19.628521 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (10ms) I0702 02:13:19.637198 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (9ms) I0702 02:13:19.737461 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0702 02:13:19.813860 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:13:19.836798 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (99ms) I0702 02:13:19.936507 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0702 02:13:20.014454 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:13:20.036278 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0702 02:13:20.137239 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (101ms) I0702 02:13:20.237329 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0702 02:13:20.336800 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (99ms) I0702 02:13:20.436917 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0702 02:13:20.536887 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0702 02:13:20.637145 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0702 02:13:20.737464 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0702 02:13:20.836459 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0702 02:13:20.937045 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0702 02:13:21.037305 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0702 02:13:21.136923 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0702 02:13:21.240305 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (103ms) I0702 02:13:21.336262 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (96ms) I0702 02:13:21.438384 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (102ms) I0702 02:13:21.537610 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (99ms) I0702 02:13:21.640042 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (102ms) I0702 02:13:21.737319 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (97ms) I0702 02:13:21.737348 1 tasks_processing.go:74] worker 1 stopped. E0702 02:13:21.737358 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0702 02:13:21.737674 1 recorder.go:75] Recording config/workload_info with fingerprint=972dad0c0463de988e62458406a486bfd291b90ce3aa6cf0b7e6605cbb9f1373 I0702 02:13:21.737689 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.240481577s to process 1 records E0702 02:13:21.737714 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.24s with: function \"workload_info\" failed with an error" I0702 02:13:21.738820 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0702 02:13:21.738833 1 periodic.go:209] Running conditional gatherer I0702 02:13:21.757197 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0702 02:13:21.763782 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.130.0.11:44700->172.30.0.10:53: read: connection refused E0702 02:13:21.764047 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 02:13:21.764109 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0702 02:13:21.776475 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0702 02:13:21.776490 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776495 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776498 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776502 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776505 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776508 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776511 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776514 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 02:13:21.776517 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0702 02:13:21.776530 1 tasks_processing.go:45] number of workers: 3 I0702 02:13:21.776540 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 02:13:21.776543 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0702 02:13:21.776552 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:13:21.776566 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0702 02:13:21.776571 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:13:21.776575 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0702 02:13:21.776579 1 tasks_processing.go:74] worker 1 stopped. I0702 02:13:21.776650 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0702 02:13:21.776664 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 958ns to process 1 records I0702 02:13:21.776702 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0702 02:13:21.776710 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.495µs to process 1 records I0702 02:13:21.776716 1 tasks_processing.go:74] worker 0 stopped. I0702 02:13:21.776829 1 tasks_processing.go:74] worker 2 stopped. I0702 02:13:21.776841 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 248.127µs to process 0 records I0702 02:13:21.776860 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.130.0.11:44700->172.30.0.10:53: read: connection refused I0702 02:13:21.776877 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0702 02:13:21.799775 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=bfeb098a498e6e6cd6f2311df936ec81bc6f891d5b7b08fb8741de60f2fe7e54 I0702 02:13:21.799885 1 diskrecorder.go:70] Writing 103 records to /var/lib/insights-operator/insights-2026-07-02-021321.tar.gz I0702 02:13:21.806430 1 diskrecorder.go:51] Wrote 103 records to disk in 6ms I0702 02:13:21.806464 1 periodic.go:278] Gathering cluster info every 2h0m0s I0702 02:13:21.806484 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0702 02:13:31.353534 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:14:34.962088 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="39d64fee94a804f6f8a0a7013c55406f6a964260f505e529995aeac099c07a38") W0702 02:14:34.962122 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0702 02:14:34.962170 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0702 02:14:34.962198 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="38bffeb0b5f986b69c5bf515557d38e1e51a7571281381b336830b96e70a8ff1") I0702 02:14:34.962278 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped"