W0702 11:29:48.425282 1 cmd.go:257] Using insecure, self-signed certificates I0702 11:29:48.624279 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:29:48.624737 1 observer_polling.go:159] Starting file observer I0702 11:29:49.394560 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0702 11:29:49.394839 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0702 11:29:49.395549 1 secure_serving.go:57] Forcing use of http/1.1 only W0702 11:29:49.395603 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0702 11:29:49.395614 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0702 11:29:49.395620 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0702 11:29:49.395624 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0702 11:29:49.395627 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0702 11:29:49.395631 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0702 11:29:49.396001 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0702 11:29:49.401008 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0702 11:29:49.401131 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"64e4b75f-77cf-45ca-ba69-5bdc5bc39959", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0702 11:29:49.404312 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0702 11:29:49.404330 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0702 11:29:49.404374 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0702 11:29:49.404462 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 11:29:49.404352 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0702 11:29:49.404547 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 11:29:49.404839 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3789556926/tls.crt::/tmp/serving-cert-3789556926/tls.key" I0702 11:29:49.405005 1 secure_serving.go:213] Serving securely on [::]:8443 I0702 11:29:49.405077 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0702 11:29:49.411426 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0702 11:29:49.411462 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 11:29:49.411549 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0702 11:29:49.417858 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 11:29:49.417880 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 11:29:49.428336 1 secretconfigobserver.go:119] support secret does not exist I0702 11:29:49.436889 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 11:29:49.442957 1 secretconfigobserver.go:119] support secret does not exist I0702 11:29:49.445588 1 recorder.go:161] Pruning old reports every 8h12m25s, max age is 288h0m0s I0702 11:29:49.455078 1 periodic.go:209] Running clusterconfig gatherer I0702 11:29:49.455187 1 tasks_processing.go:45] number of workers: 32 I0702 11:29:49.455249 1 tasks_processing.go:69] worker 31 listening for tasks. I0702 11:29:49.455270 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0702 11:29:49.455351 1 tasks_processing.go:69] worker 12 listening for tasks. I0702 11:29:49.455428 1 tasks_processing.go:69] worker 4 listening for tasks. I0702 11:29:49.455443 1 tasks_processing.go:71] worker 4 working on openstack_dataplanenodesets task. I0702 11:29:49.455463 1 tasks_processing.go:69] worker 13 listening for tasks. I0702 11:29:49.455514 1 tasks_processing.go:69] worker 14 listening for tasks. I0702 11:29:49.455542 1 tasks_processing.go:69] worker 15 listening for tasks. I0702 11:29:49.455573 1 tasks_processing.go:69] worker 16 listening for tasks. I0702 11:29:49.455602 1 tasks_processing.go:69] worker 17 listening for tasks. I0702 11:29:49.455629 1 tasks_processing.go:69] worker 18 listening for tasks. I0702 11:29:49.455660 1 tasks_processing.go:69] worker 19 listening for tasks. I0702 11:29:49.455689 1 tasks_processing.go:69] worker 20 listening for tasks. I0702 11:29:49.455712 1 tasks_processing.go:69] worker 21 listening for tasks. I0702 11:29:49.455724 1 tasks_processing.go:69] worker 22 listening for tasks. I0702 11:29:49.455731 1 tasks_processing.go:69] worker 28 listening for tasks. I0702 11:29:49.455742 1 tasks_processing.go:69] worker 29 listening for tasks. I0702 11:29:49.455752 1 tasks_processing.go:69] worker 6 listening for tasks. I0702 11:29:49.455758 1 tasks_processing.go:69] worker 30 listening for tasks. I0702 11:29:49.455253 1 tasks_processing.go:69] worker 7 listening for tasks. I0702 11:29:49.455765 1 tasks_processing.go:69] worker 5 listening for tasks. I0702 11:29:49.455277 1 tasks_processing.go:71] worker 31 working on install_plans task. I0702 11:29:49.455778 1 tasks_processing.go:69] worker 24 listening for tasks. I0702 11:29:49.455787 1 tasks_processing.go:69] worker 23 listening for tasks. I0702 11:29:49.455799 1 tasks_processing.go:69] worker 25 listening for tasks. I0702 11:29:49.455810 1 tasks_processing.go:69] worker 26 listening for tasks. I0702 11:29:49.455818 1 tasks_processing.go:71] worker 13 working on feature_gates task. I0702 11:29:49.455823 1 tasks_processing.go:71] worker 26 working on version task. I0702 11:29:49.456055 1 tasks_processing.go:71] worker 22 working on container_images task. I0702 11:29:49.456384 1 tasks_processing.go:71] worker 28 working on lokistack task. I0702 11:29:49.455431 1 tasks_processing.go:71] worker 12 working on tsdb_status task. W0702 11:29:49.456448 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:29:49.456469 1 tasks_processing.go:71] worker 12 working on image_registries task. I0702 11:29:49.456700 1 tasks_processing.go:71] worker 14 working on sap_config task. I0702 11:29:49.456738 1 tasks_processing.go:71] worker 6 working on openstack_controlplanes task. I0702 11:29:49.455348 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0702 11:29:49.456981 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 37.641µs to process 0 records I0702 11:29:49.455286 1 tasks_processing.go:69] worker 8 listening for tasks. I0702 11:29:49.457001 1 tasks_processing.go:71] worker 8 working on ceph_cluster task. I0702 11:29:49.457010 1 tasks_processing.go:71] worker 30 working on sap_pods task. I0702 11:29:49.457164 1 tasks_processing.go:71] worker 15 working on node_logs task. I0702 11:29:49.455295 1 tasks_processing.go:69] worker 9 listening for tasks. I0702 11:29:49.455303 1 tasks_processing.go:69] worker 10 listening for tasks. I0702 11:29:49.455310 1 tasks_processing.go:69] worker 11 listening for tasks. I0702 11:29:49.455389 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0702 11:29:49.457222 1 insightsreport.go:296] Starting report retriever I0702 11:29:49.457230 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0702 11:29:49.455400 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 11:29:49.455408 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 11:29:49.455415 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 11:29:49.455421 1 tasks_processing.go:69] worker 3 listening for tasks. I0702 11:29:49.457249 1 tasks_processing.go:71] worker 3 working on image_pruners task. I0702 11:29:49.457472 1 tasks_processing.go:71] worker 9 working on machine_configs task. I0702 11:29:49.457729 1 tasks_processing.go:71] worker 7 working on image task. I0702 11:29:49.457753 1 tasks_processing.go:71] worker 24 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0702 11:29:49.457818 1 tasks_processing.go:71] worker 10 working on proxies task. I0702 11:29:49.457965 1 tasks_processing.go:71] worker 11 working on storage_cluster task. I0702 11:29:49.458013 1 tasks_processing.go:71] worker 2 working on support_secret task. I0702 11:29:49.458050 1 tasks_processing.go:71] worker 25 working on networks task. I0702 11:29:49.457986 1 tasks_processing.go:69] worker 27 listening for tasks. I0702 11:29:49.457733 1 tasks_processing.go:71] worker 16 working on silenced_alerts task. I0702 11:29:49.458188 1 tasks_processing.go:71] worker 27 working on machine_sets task. W0702 11:29:49.458202 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:29:49.458217 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 21.992µs to process 0 records I0702 11:29:49.457740 1 tasks_processing.go:71] worker 18 working on machine_config_pools task. I0702 11:29:49.457737 1 tasks_processing.go:71] worker 17 working on config_maps task. I0702 11:29:49.457742 1 tasks_processing.go:71] worker 19 working on crds task. I0702 11:29:49.457744 1 tasks_processing.go:71] worker 20 working on monitoring_persistent_volumes task. I0702 11:29:49.457745 1 tasks_processing.go:71] worker 21 working on authentication task. I0702 11:29:49.457749 1 tasks_processing.go:71] worker 23 working on infrastructures task. I0702 11:29:49.456732 1 tasks_processing.go:71] worker 29 working on validating_webhook_configurations task. I0702 11:29:49.458002 1 tasks_processing.go:71] worker 1 working on pod_network_connectivity_checks task. I0702 11:29:49.457751 1 tasks_processing.go:71] worker 5 working on ingress_certificates task. I0702 11:29:49.458401 1 tasks_processing.go:71] worker 16 working on storage_classes task. I0702 11:29:49.458006 1 tasks_processing.go:71] worker 0 working on ingress task. I0702 11:29:49.461526 1 tasks_processing.go:71] worker 4 working on active_alerts task. W0702 11:29:49.461559 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:29:49.461572 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 6.05335ms to process 0 records I0702 11:29:49.461580 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 27.74µs to process 0 records I0702 11:29:49.461586 1 tasks_processing.go:71] worker 4 working on aggregated_monitoring_cr_names task. I0702 11:29:49.461650 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 4.904735ms to process 0 records I0702 11:29:49.461660 1 tasks_processing.go:71] worker 14 working on openshift_machine_api_events task. I0702 11:29:49.462002 1 tasks_processing.go:71] worker 6 working on qemu_kubevirt_launcher_logs task. I0702 11:29:49.462016 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 5.248695ms to process 0 records I0702 11:29:49.462209 1 tasks_processing.go:71] worker 28 working on dvo_metrics task. I0702 11:29:49.462221 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 5.801771ms to process 0 records I0702 11:29:49.463139 1 tasks_processing.go:71] worker 30 working on operators task. I0702 11:29:49.463195 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 6.119303ms to process 0 records I0702 11:29:49.463402 1 tasks_processing.go:71] worker 8 working on olm_operators task. I0702 11:29:49.463410 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 6.387143ms to process 0 records I0702 11:29:49.468919 1 tasks_processing.go:71] worker 1 working on mutating_webhook_configurations task. E0702 11:29:49.468979 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0702 11:29:49.469569 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 9.846072ms to process 0 records I0702 11:29:49.474802 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0702 11:29:49.474825 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 11:29:49.474830 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 11:29:49.474835 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 11:29:49.474865 1 controller.go:489] The operator is still being initialized I0702 11:29:49.474875 1 controller.go:512] The operator is healthy I0702 11:29:49.475039 1 tasks_processing.go:71] worker 3 working on schedulers task. I0702 11:29:49.475582 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=e434af7f10b274b599c2f79e7a6575d6b1e17899984bbd9afdc9416c84cfc1c7 I0702 11:29:49.475599 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 17.781196ms to process 1 records I0702 11:29:49.475852 1 tasks_processing.go:71] worker 13 working on cluster_apiserver task. I0702 11:29:49.476270 1 recorder.go:75] Recording config/featuregate with fingerprint=466bb684ae1033df292faea65e7ee6ba9e706b5a04c0b0378b99166b62ed33cc I0702 11:29:49.476283 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 20.026556ms to process 1 records I0702 11:29:49.478231 1 tasks_processing.go:71] worker 27 working on operators_pods_and_events task. I0702 11:29:49.478253 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 20.031987ms to process 0 records I0702 11:29:49.478432 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 20.229002ms to process 0 records I0702 11:29:49.478444 1 tasks_processing.go:71] worker 11 working on clusterroles task. I0702 11:29:49.487761 1 tasks_processing.go:71] worker 23 working on metrics task. W0702 11:29:49.487839 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:29:49.488546 1 recorder.go:75] Recording config/infrastructure with fingerprint=d9611efc8aad56679aa9c27713f5cd7790c4f3821309b9129bb5f6d1e1ae3cb6 I0702 11:29:49.488614 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 28.898589ms to process 1 records I0702 11:29:49.488803 1 tasks_processing.go:71] worker 21 working on machine_healthchecks task. I0702 11:29:49.488970 1 recorder.go:75] Recording config/authentication with fingerprint=a2904d5484bbd6344d3b136ec1077805409658149615dcdb070b7f99c230983e I0702 11:29:49.489016 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 29.077231ms to process 1 records I0702 11:29:49.489062 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 54.311µs to process 0 records I0702 11:29:49.489127 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 29.46554ms to process 0 records I0702 11:29:49.489114 1 tasks_processing.go:71] worker 23 working on openstack_version task. I0702 11:29:49.489158 1 tasks_processing.go:71] worker 20 working on openshift_logging task. I0702 11:29:49.489635 1 tasks_processing.go:71] worker 7 working on sap_datahubs task. I0702 11:29:49.489779 1 recorder.go:75] Recording config/image with fingerprint=966279c3aaf34adf0d5cb1f5164bba640f6559b01dce9725cae8c12918cb435b I0702 11:29:49.489801 1 gather.go:177] gatherer "clusterconfig" function "image" took 31.875036ms to process 1 records I0702 11:29:49.489862 1 recorder.go:75] Recording config/proxy with fingerprint=472c94d9335d3859a3b746474b03ffe81fb42d805040943ab28c691698941b1b I0702 11:29:49.489928 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 31.848851ms to process 1 records I0702 11:29:49.489913 1 tasks_processing.go:71] worker 10 working on openstack_dataplanedeployments task. I0702 11:29:49.491074 1 tasks_processing.go:71] worker 12 working on machines task. I0702 11:29:49.501949 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=36e75180ef296001c899ef6fa2c127a8a9e52d5a74407c8bc1b57773c9a48dfe I0702 11:29:49.501999 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 34.59649ms to process 1 records I0702 11:29:49.502028 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 30.068771ms to process 0 records I0702 11:29:49.502254 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=7256987e2fb378d53a5c7033fc2cd4ebe453611e10e4f983d79405e66563e9e2 I0702 11:29:49.502324 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=26d5ab31441ba36d8d3d9fbbd2f0e454ed0159c7373e30584826730179f457a1 I0702 11:29:49.502424 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=350cc688ed3c8ab6df5eafb8e505cf4fb8d0a617aec00133ea676cd3c13763dc I0702 11:29:49.502470 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 23.630767ms to process 3 records I0702 11:29:49.502501 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 35.49857ms to process 0 records I0702 11:29:49.502655 1 tasks_processing.go:71] worker 1 working on cost_management_metrics_configs task. I0702 11:29:49.502733 1 recorder.go:75] Recording config/network with fingerprint=a3dfd9ef31c64d0fcf6993d3d0f26f43ee14cad5f100e37bfc3ccf49a085a2a3 I0702 11:29:49.502765 1 gather.go:177] gatherer "clusterconfig" function "networks" took 35.100481ms to process 1 records E0702 11:29:49.502795 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0702 11:29:49.502822 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 35.157625ms to process 0 records I0702 11:29:49.502936 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=f2ebb74aa1c9c9f474de49c8af190d44bcc5b60736477e098565f85eab6f4180 I0702 11:29:49.502975 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=f24eec1b8073e8d75cd909569d1586060cbcbeb6a0c856bb8d19c147c2edb7e6 I0702 11:29:49.503004 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 37.875362ms to process 2 records I0702 11:29:49.503071 1 tasks_processing.go:71] worker 16 working on jaegers task. I0702 11:29:49.503197 1 recorder.go:75] Recording config/apiserver with fingerprint=e64c5b518f28268d74899f6b4f8c78690f8d47290dd80f875203a55ffa37641c I0702 11:29:49.503239 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 21.208789ms to process 1 records I0702 11:29:49.503416 1 recorder.go:75] Recording config/ingress with fingerprint=219d3f91221f70e194a3b16ef9ba14e58254af549e604e29b48893ef116d00a6 I0702 11:29:49.503467 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 38.040694ms to process 1 records I0702 11:29:49.503500 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 8.141226ms to process 0 records I0702 11:29:49.503525 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 8.874345ms to process 0 records I0702 11:29:49.503016 1 tasks_processing.go:71] worker 14 working on nodenetworkconfigurationpolicies task. I0702 11:29:49.503586 1 tasks_processing.go:71] worker 18 working on oauths task. I0702 11:29:49.503024 1 tasks_processing.go:71] worker 25 working on nodes task. I0702 11:29:49.503029 1 tasks_processing.go:71] worker 15 working on container_runtime_configs task. I0702 11:29:49.503870 1 tasks_processing.go:71] worker 20 working on nodenetworkstates task. I0702 11:29:49.503099 1 tasks_processing.go:71] worker 2 working on pdbs task. I0702 11:29:49.504125 1 tasks_processing.go:71] worker 13 working on machine_autoscalers task. I0702 11:29:49.503552 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 39.934271ms to process 0 records I0702 11:29:49.504157 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 8.537867ms to process 0 records I0702 11:29:49.504174 1 tasks_processing.go:71] worker 0 working on overlapping_namespace_uids task. I0702 11:29:49.504246 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=a630bc54f9cc7ce47c9eced63a67a0ad78f0f9a939d085086f34440d3babb36b I0702 11:29:49.504299 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 23.864715ms to process 1 records I0702 11:29:49.504329 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 38.875897ms to process 0 records E0702 11:29:49.504355 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0702 11:29:49.504401 1 tasks_processing.go:74] worker 21 stopped. I0702 11:29:49.504335 1 tasks_processing.go:71] worker 3 working on certificate_signing_requests task. I0702 11:29:49.504408 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 13.79609ms to process 0 records I0702 11:29:49.504616 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 11:29:49.504696 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=6d2f816541dd07c677aabfead4243f7d6fe53363f4c0d9a2d593d223830254da I0702 11:29:49.504851 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=82fe4cf776240a6983ffe0d09129f455466d3f26fc4081d3a345bf4205aa0810 I0702 11:29:49.504888 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=d5aae0e562ba3eb9e93f97accb61a270f8b002a52601281c609eb1a16c4c310a I0702 11:29:49.504932 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=cb37ab4ac87e448c2e017818a668b48cef41ddaabe1b1e0fa03ec65cdb893682 I0702 11:29:49.504944 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0702 11:29:49.504974 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=8309d78a0ae73e1e4bc62258d67ef48bda79c7e687ad2ff5bfa9a712572b476e I0702 11:29:49.505016 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=a5132e09a1a289425e3ba76186e610cd473ae74ff02d59ba9a1d24f1cfef45d5 W0702 11:29:49.505029 1 operator.go:288] started I0702 11:29:49.505054 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0702 11:29:49.505062 1 tasks_processing.go:74] worker 29 stopped. I0702 11:29:49.504383 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0702 11:29:49.505083 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0702 11:29:49.504342 1 tasks_processing.go:74] worker 7 stopped. I0702 11:29:49.504313 1 tasks_processing.go:71] worker 10 working on service_accounts task. I0702 11:29:49.504392 1 tasks_processing.go:74] worker 8 stopped. I0702 11:29:49.505056 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=6e7691d9a41cbdf36fdcd47439073746040bcef3d8f419299daa91b65c973602 I0702 11:29:49.505617 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=788e0eb9e9f408765b9f079a479285e02234e9d09b0e2f6ffeefddfc116610a0 I0702 11:29:49.505675 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=fd828fecebcfffe3c3d5f79d194ee1a3a619ee18760efc577af3f16d713599bb I0702 11:29:49.505721 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=726d699eb287dd85334798b31fa2cc0047f8b60a1a574b51a826b5c7f8d137bd I0702 11:29:49.505757 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=539dac0057f39aee329d4798f54128ecb003f8d4fb6891489e90022e5234c941 I0702 11:29:49.505766 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 43.69968ms to process 11 records I0702 11:29:49.505774 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 14.171363ms to process 0 records I0702 11:29:49.505677 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 11:29:49.505914 1 tasks_processing.go:74] worker 23 stopped. I0702 11:29:49.505995 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=4106dd3aa3264c3118d6feafe8a3f9f2c3a30f1320edd47ac171158eed6ec01e I0702 11:29:49.506142 1 tasks_processing.go:74] worker 11 stopped. I0702 11:29:49.506143 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=1e45b93ae2518a8ed34af88b7e0e2597098b8eb95a406d5562677e337017c92a I0702 11:29:49.506223 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 26.030288ms to process 2 records E0702 11:29:49.506251 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0702 11:29:49.506821 1 gather.go:177] gatherer "clusterconfig" function "machines" took 13.310591ms to process 0 records I0702 11:29:49.506352 1 tasks_processing.go:74] worker 12 stopped. I0702 11:29:49.507038 1 tasks_processing.go:74] worker 4 stopped. I0702 11:29:49.507049 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 45.442572ms to process 0 records I0702 11:29:49.512097 1 base_controller.go:82] Caches are synced for ConfigController I0702 11:29:49.512141 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0702 11:29:49.513145 1 tasks_processing.go:74] worker 1 stopped. I0702 11:29:49.513464 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 10.470849ms to process 0 records I0702 11:29:49.515290 1 tasks_processing.go:74] worker 16 stopped. I0702 11:29:49.515309 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 12.161989ms to process 0 records I0702 11:29:49.515345 1 tasks_processing.go:74] worker 15 stopped. I0702 11:29:49.515358 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 11.568079ms to process 0 records I0702 11:29:49.515407 1 tasks_processing.go:74] worker 14 stopped. I0702 11:29:49.515590 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 11.845174ms to process 0 records I0702 11:29:49.515639 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 11.646287ms to process 0 records I0702 11:29:49.515647 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 11.408148ms to process 0 records I0702 11:29:49.515654 1 tasks_processing.go:74] worker 13 stopped. I0702 11:29:49.515665 1 tasks_processing.go:74] worker 20 stopped. W0702 11:29:49.516902 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 11:29:49.517930 1 tasks_processing.go:74] worker 22 stopped. I0702 11:29:49.519432 1 recorder.go:75] Recording config/pod/openshift-multus/multus-s2qqw with fingerprint=dbfd6840060c06994a162b742e2751416ff6d99d4e81759ea73b846a8e25c7e1 I0702 11:29:49.519736 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-vltxz with fingerprint=8cee26f3f1a65d42f7fc6c015fd37033fef16b0814dee2cd086802121c28bb16 I0702 11:29:49.519782 1 recorder.go:75] Recording config/running_containers with fingerprint=7bd8414aae45ecba6b1f9ba0c5881ddbdb42756d44fb0abe929b0c6d7bdf27e8 I0702 11:29:49.519792 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 61.815023ms to process 3 records I0702 11:29:49.519914 1 tasks_processing.go:74] worker 19 stopped. I0702 11:29:49.520225 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=f24d6b02341741ce9b754baaeca8c84256cc43b2e20a5f8e94cc6546ac46e124 I0702 11:29:49.520536 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=cd8ee66eeb46709b7263c6864248df2c438dd85dc2585ae0b177d4fde7040f46 I0702 11:29:49.520588 1 gather.go:177] gatherer "clusterconfig" function "crds" took 60.273988ms to process 2 records I0702 11:29:49.529344 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 11:29:49.529460 1 prometheus_rules.go:88] Prometheus rules successfully created I0702 11:29:49.529827 1 tasks_processing.go:74] worker 18 stopped. I0702 11:29:49.529924 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 11:29:49.529940 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 11:29:49.529944 1 controller.go:212] Source scaController *sca.Controller is not ready I0702 11:29:49.529948 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0702 11:29:49.529951 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 11:29:49.529966 1 controller.go:489] The operator is still being initialized I0702 11:29:49.529972 1 controller.go:512] The operator is healthy I0702 11:29:49.530268 1 recorder.go:75] Recording config/oauth with fingerprint=402d76e5f67df6219e8b366bcc28f320d9e18466f384f16eccecb0ac7304f7c7 I0702 11:29:49.530314 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 26.212758ms to process 1 records I0702 11:29:49.530827 1 tasks_processing.go:74] worker 2 stopped. I0702 11:29:49.530909 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=34c9da408410eb3008067c8c518612755d8fd17d6b2f23f032ae175b09b2a255 I0702 11:29:49.530929 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=5cfd5113acc3225997cdf7909f5009512ab5d48f7558fbd0a9a51a26c96eb54f I0702 11:29:49.530944 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=a4e8dae50655a69d27b3d124ed82f976783cc3085b73da7b24776b5c82b78f9f I0702 11:29:49.530953 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 26.794428ms to process 3 records I0702 11:29:49.532657 1 tasks_processing.go:74] worker 3 stopped. I0702 11:29:49.532712 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 28.168503ms to process 0 records E0702 11:29:49.539265 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27da50040b-9adc-48b3-9ba3-305441ce95ee%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.16:47201->172.30.0.10:53: read: connection refused I0702 11:29:49.539317 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27da50040b-9adc-48b3-9ba3-305441ce95ee%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.16:47201->172.30.0.10:53: read: connection refused I0702 11:29:49.548773 1 tasks_processing.go:74] worker 17 stopped. E0702 11:29:49.548789 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0702 11:29:49.548797 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0702 11:29:49.548801 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0702 11:29:49.548810 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0702 11:29:49.548872 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0702 11:29:49.548883 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0702 11:29:49.548889 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0702 11:29:49.548895 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0702 11:29:49.548950 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0702 11:29:49.548961 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0702 11:29:49.548970 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 90.347269ms to process 7 records I0702 11:29:49.549143 1 tasks_processing.go:74] worker 5 stopped. E0702 11:29:49.549156 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0702 11:29:49.549161 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2r9vhpln4uc0l5dn3a1fjcopb8uih7bq-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2r9vhpln4uc0l5dn3a1fjcopb8uih7bq-primary-cert-bundle-secret" not found I0702 11:29:49.549210 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=1437103693501e45312682db8b38eb846c0f4ae2878e9a42675511ddc1f5b13c I0702 11:29:49.549225 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 90.044932ms to process 1 records I0702 11:29:49.569873 1 tasks_processing.go:74] worker 26 stopped. I0702 11:29:49.569907 1 gather_logs.go:145] no pods in namespace were found I0702 11:29:49.570298 1 recorder.go:75] Recording config/version with fingerprint=93e2e598c07ad2873de7ac7125fe112dbf9842275820deda6ec379facce0750b I0702 11:29:49.570314 1 recorder.go:75] Recording config/id with fingerprint=86ee5a29dcdf202eb68349b8217a49df5ade8ddcc1facc0f92c389467e05e77b I0702 11:29:49.570324 1 gather.go:177] gatherer "clusterconfig" function "version" took 114.039792ms to process 2 records I0702 11:29:49.570339 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 107.939946ms to process 0 records I0702 11:29:49.570349 1 tasks_processing.go:74] worker 6 stopped. I0702 11:29:49.576223 1 tasks_processing.go:74] worker 25 stopped. I0702 11:29:49.576695 1 recorder.go:75] Recording config/node/ip-10-0-0-43.ec2.internal with fingerprint=d39d4f1ba17a04addad5e9eb9f6acdc611298eab560c9cbe53447ad438bfc9bb I0702 11:29:49.576853 1 recorder.go:75] Recording config/node/ip-10-0-1-132.ec2.internal with fingerprint=eac4bfffc59cf8fa7cef8ddd4a473af55f1ae76f5b104771040e654a2fac7b18 I0702 11:29:49.576974 1 recorder.go:75] Recording config/node/ip-10-0-2-8.ec2.internal with fingerprint=3165f0876989663bc083bcf8cbb47f2053efa919a979f9e271322d155bba7f98 I0702 11:29:49.577010 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 72.456055ms to process 3 records I0702 11:29:49.581712 1 tasks_processing.go:74] worker 9 stopped. I0702 11:29:49.581803 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0702 11:29:49.581839 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 124.220146ms to process 1 records I0702 11:29:49.584628 1 tasks_processing.go:74] worker 0 stopped. I0702 11:29:49.584665 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0702 11:29:49.584677 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 80.396144ms to process 1 records I0702 11:29:49.590250 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0702 11:29:49.592144 1 tasks_processing.go:74] worker 24 stopped. I0702 11:29:49.592200 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 134.380917ms to process 0 records W0702 11:29:49.596517 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.16:55732->172.30.0.10:53: read: connection refused I0702 11:29:49.596585 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.16:55732->172.30.0.10:53: read: connection refused I0702 11:29:49.605295 1 base_controller.go:82] Caches are synced for LoggingSyncer I0702 11:29:49.605314 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0702 11:29:49.908312 1 gather_cluster_operator_pods_and_events.go:121] Found 18 pods with 21 containers I0702 11:29:49.908335 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1198372 bytes I0702 11:29:49.908582 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-gtd5j pod in namespace openshift-dns (previous: false). I0702 11:29:50.136637 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gtd5j pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-gtd5j\" is waiting to start: ContainerCreating" I0702 11:29:50.136661 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-gtd5j\" is waiting to start: ContainerCreating" I0702 11:29:50.136673 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-gtd5j pod in namespace openshift-dns (previous: false). I0702 11:29:50.313294 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gtd5j pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-gtd5j\" is waiting to start: ContainerCreating" I0702 11:29:50.313316 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-gtd5j\" is waiting to start: ContainerCreating" I0702 11:29:50.313328 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-hkhsl pod in namespace openshift-dns (previous: false). W0702 11:29:50.502346 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 11:29:50.532645 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-hkhsl pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-hkhsl\" is waiting to start: ContainerCreating" I0702 11:29:50.532685 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-hkhsl\" is waiting to start: ContainerCreating" I0702 11:29:50.532697 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-hkhsl pod in namespace openshift-dns (previous: false). I0702 11:29:50.711320 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-hkhsl pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-hkhsl\" is waiting to start: ContainerCreating" I0702 11:29:50.711338 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-hkhsl\" is waiting to start: ContainerCreating" I0702 11:29:50.711351 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-rp9ps pod in namespace openshift-dns (previous: false). I0702 11:29:50.931619 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0702 11:29:50.947159 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-rp9ps pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-rp9ps\" is waiting to start: ContainerCreating" I0702 11:29:50.947181 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-rp9ps\" is waiting to start: ContainerCreating" I0702 11:29:50.947192 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-rp9ps pod in namespace openshift-dns (previous: false). I0702 11:29:51.122708 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-rp9ps pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-rp9ps\" is waiting to start: ContainerCreating" I0702 11:29:51.122725 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-rp9ps\" is waiting to start: ContainerCreating" I0702 11:29:51.122736 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-kgqln pod in namespace openshift-dns (previous: false). I0702 11:29:51.316335 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:51.316353 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-n5sr4 pod in namespace openshift-dns (previous: false). I0702 11:29:51.337834 1 tasks_processing.go:74] worker 30 stopped. I0702 11:29:51.337890 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=0f01a58b9ba2103c3f2a860e9f847f4d3408344bfe54feeb1c36a685e4279c3e I0702 11:29:51.337933 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=392657db5135f9a3bef2b12f9f365f99837ec56c7706716ca081f9e206e21046 I0702 11:29:51.337960 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0702 11:29:51.337987 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=6a511437b9b48e64f556369045448d52d4ffbe89442cd06a8aa26a89cc67f9af I0702 11:29:51.338022 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0702 11:29:51.338048 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=d0c93198349f8d87a8fc02e77fcde5e7efcf7045513aa6c29958de06124f565c I0702 11:29:51.338092 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=bd346527d7f5beb4f069d447ae66ac11451b7a00b1fc7e465e03c18fe2a3c7fb I0702 11:29:51.338118 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=6e705de887346028a00acd609fc16d334ab15e1d373678532b180bc3e8654147 I0702 11:29:51.338132 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=d0f0b2139fbef051cb81d8e3bd01eee0f4616e78fe5ac2b0498ebec495a159bc I0702 11:29:51.338150 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=2ee714cdff3ccd279f1f43255a155eb9643eea62698d83ea9adcd8b7f7298d87 I0702 11:29:51.338164 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0702 11:29:51.338182 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=33768adb4ae08bf6b7933cbd578e22c6695d909dacb49c44c88d76891540340c I0702 11:29:51.338192 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0702 11:29:51.338207 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=c1d6660eb61eae63d7329a4bf2e8b97e95b7117f145b6a201eb0cc1bdfcccc53 I0702 11:29:51.338214 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0702 11:29:51.338227 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=b0a1608986d7b1cc35d46c46fa8f15c43d45da2bc002da05d0428441ba0b209a I0702 11:29:51.338237 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0702 11:29:51.338254 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=6a0cbab7a3a5cb97cfd040dfbb0e71d149ccbc6b2a9fc465b41285cacdbe2006 I0702 11:29:51.338406 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=7089a6a5b361e1bf55a3337767408a0ac837636c8bb19906cf622939b0cf5218 I0702 11:29:51.338419 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0702 11:29:51.338429 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0702 11:29:51.338467 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0702 11:29:51.338490 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=869269ca115d9739f0aa3f84798ccbb86b97087ba2ef59cc2015d6fe29b77e64 I0702 11:29:51.338515 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=1cd8ab21866621375ededeaebd6e44baf822a92ae8b3040d68ed769c35bc0409 I0702 11:29:51.338523 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0702 11:29:51.338547 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=337d21ef4f7827aad54ddeea7f5cfe9ab957b928b56e2f2bd739efcb9f49fecd I0702 11:29:51.338566 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0702 11:29:51.338580 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=686dc9706b414c88519ba57ea0a6401c5ede6495cc9382e48b1775d6173d7f3c I0702 11:29:51.338593 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=3ad96436615b12fca4d1b5761bbdfc858fa48f9f552adb6c922e137d5d365f2f I0702 11:29:51.338607 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=fe3110c07f74de2631a794b41c8ed74fae894d535c1a51533ef8ef8a4f1e1647 I0702 11:29:51.338626 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=d29364f46d5190eb01dcad6265bdad5a9566cf6d29827db6b18474856e0c7acc I0702 11:29:51.338640 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=9bf68cfe0882d5dfa130e68cc54ab015dd48c1f83627f7cef2519593f90672b6 I0702 11:29:51.338668 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=9219f506d98bbb02fec3e5f164605153c35db401c7dcbd26bbfbb0464ea18dc4 I0702 11:29:51.338683 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0702 11:29:51.338692 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0702 11:29:51.338702 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.874669261s to process 35 records W0702 11:29:51.501310 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 11:29:51.527225 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:51.527249 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-q2w8l pod in namespace openshift-dns (previous: false). I0702 11:29:51.721803 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:51.721823 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-686c55544d-n67ld pod in namespace openshift-image-registry (previous: false). I0702 11:29:51.912702 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-686c55544d-n67ld pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-686c55544d-n67ld\" is waiting to start: ContainerCreating" I0702 11:29:51.912719 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-686c55544d-n67ld\" is waiting to start: ContainerCreating" I0702 11:29:51.912730 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-dc99b9799-llvpf pod in namespace openshift-image-registry (previous: false). I0702 11:29:52.114181 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-dc99b9799-llvpf pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-dc99b9799-llvpf\" is waiting to start: ContainerCreating" I0702 11:29:52.114198 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-dc99b9799-llvpf\" is waiting to start: ContainerCreating" I0702 11:29:52.114209 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-dc99b9799-qtqwd pod in namespace openshift-image-registry (previous: false). I0702 11:29:52.312189 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-dc99b9799-qtqwd pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-dc99b9799-qtqwd\" is waiting to start: ContainerCreating" I0702 11:29:52.312207 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-dc99b9799-qtqwd\" is waiting to start: ContainerCreating" I0702 11:29:52.312217 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-9d9lr pod in namespace openshift-image-registry (previous: false). W0702 11:29:52.500509 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 11:29:52.516534 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:52.516550 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-gsn5c pod in namespace openshift-image-registry (previous: false). I0702 11:29:52.712107 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:52.712127 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-vd7hc pod in namespace openshift-image-registry (previous: false). I0702 11:29:52.919840 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0702 11:29:52.919859 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5d5887bb94-q5fw2 pod in namespace openshift-ingress (previous: false). I0702 11:29:53.113973 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5d5887bb94-q5fw2 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5d5887bb94-q5fw2\" is waiting to start: ContainerCreating" I0702 11:29:53.113996 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5d5887bb94-q5fw2\" is waiting to start: ContainerCreating" I0702 11:29:53.114010 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5d5887bb94-xrczn pod in namespace openshift-ingress (previous: false). I0702 11:29:53.313203 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5d5887bb94-xrczn pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5d5887bb94-xrczn\" is waiting to start: ContainerCreating" I0702 11:29:53.313224 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5d5887bb94-xrczn\" is waiting to start: ContainerCreating" I0702 11:29:53.313239 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-85bb8945-qcddl pod in namespace openshift-ingress (previous: false). W0702 11:29:53.501090 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 11:29:53.512881 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-85bb8945-qcddl pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-85bb8945-qcddl\" is waiting to start: ContainerCreating" I0702 11:29:53.512904 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-85bb8945-qcddl\" is waiting to start: ContainerCreating" I0702 11:29:53.512917 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-64d54 pod in namespace openshift-ingress-canary (previous: false). I0702 11:29:53.712889 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-64d54 pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-64d54\" is waiting to start: ContainerCreating" I0702 11:29:53.712910 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-64d54\" is waiting to start: ContainerCreating" I0702 11:29:53.712923 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-k976b pod in namespace openshift-ingress-canary (previous: false). I0702 11:29:53.912958 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-k976b pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-k976b\" is waiting to start: ContainerCreating" I0702 11:29:53.912977 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-k976b\" is waiting to start: ContainerCreating" I0702 11:29:53.912988 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-wcvzz pod in namespace openshift-ingress-canary (previous: false). I0702 11:29:54.112033 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-wcvzz pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-wcvzz\" is waiting to start: ContainerCreating" I0702 11:29:54.112052 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-wcvzz\" is waiting to start: ContainerCreating" I0702 11:29:54.112067 1 tasks_processing.go:74] worker 27 stopped. I0702 11:29:54.112144 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=e4bc0b13c4a2c8050e64ddb8692637f0763c8ab36ecac963855869276336eb93 I0702 11:29:54.112196 1 recorder.go:75] Recording events/openshift-dns with fingerprint=52c855d8ab9b00a3ff891f9cadc3b8f61fd6dfc37e2a3902d25e8a2a2502e644 I0702 11:29:54.112294 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=bf753221b0ac539bf0d4e6d2d51f00f08a3968e3879bec91d61f90984cc911b4 I0702 11:29:54.112320 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=1fd1aa5f963cc516cc638375a72ceec59cdb5962bd3af8d6a84be1ea12835cd5 I0702 11:29:54.112379 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=9c45d38a05d9fcb00298cff1748009ee373149193b955bf000ffac21303482eb I0702 11:29:54.112396 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=2945c51340c688ce6afabbd344782a6c894f00ea705b79c860b478213352e3ad I0702 11:29:54.112405 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.633751973s to process 6 records W0702 11:29:54.500509 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0702 11:29:54.500551 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 11:29:54.500570 1 tasks_processing.go:74] worker 28 stopped. E0702 11:29:54.500583 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 11:29:54.500598 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0702 11:29:54.500620 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0702 11:29:54.500636 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.038303011s to process 1 records I0702 11:30:02.101954 1 tasks_processing.go:74] worker 31 stopped. I0702 11:30:02.102001 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0702 11:30:02.102017 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.646163406s to process 1 records I0702 11:30:02.688123 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 11:30:02.912325 1 tasks_processing.go:74] worker 10 stopped. I0702 11:30:02.912618 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=6eea9bca9c2d016e4128bf2b374fefdd9610f4d08552d21add7f6bb399a3af40 I0702 11:30:02.912636 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.407179543s to process 1 records E0702 11:30:02.912697 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.457s with: function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0702 11:30:02.913797 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0702 11:30:02.913812 1 periodic.go:209] Running workloads gatherer I0702 11:30:02.913825 1 tasks_processing.go:45] number of workers: 2 I0702 11:30:02.913831 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 11:30:02.913835 1 tasks_processing.go:71] worker 1 working on workload_info task. I0702 11:30:02.913842 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 11:30:02.913859 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0702 11:30:02.952219 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0702 11:30:02.960269 1 tasks_processing.go:74] worker 0 stopped. I0702 11:30:02.960286 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 46.393787ms to process 0 records I0702 11:30:02.971200 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (20ms) I0702 11:30:02.991531 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (20ms) I0702 11:30:03.003586 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (12ms) I0702 11:30:03.014397 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (11ms) I0702 11:30:03.024812 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (10ms) I0702 11:30:03.035495 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (11ms) I0702 11:30:03.045758 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (10ms) I0702 11:30:03.061504 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (16ms) I0702 11:30:03.070947 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (9ms) I0702 11:30:03.080927 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (10ms) I0702 11:30:03.090696 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (10ms) I0702 11:30:03.162438 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (72ms) I0702 11:30:03.263354 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (101ms) I0702 11:30:03.362538 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0702 11:30:03.462994 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0702 11:30:03.567590 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (105ms) I0702 11:30:03.662656 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (95ms) I0702 11:30:03.762547 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (100ms) I0702 11:30:03.863640 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (101ms) I0702 11:30:03.962441 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (99ms) I0702 11:30:04.062464 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0702 11:30:04.162830 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0702 11:30:04.263605 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (101ms) I0702 11:30:04.362667 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (99ms) I0702 11:30:04.462626 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0702 11:30:04.562127 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (99ms) I0702 11:30:04.663460 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (101ms) I0702 11:30:04.762590 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0702 11:30:04.862602 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0702 11:30:04.962342 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0702 11:30:05.063122 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0702 11:30:05.162647 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0702 11:30:05.263310 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (101ms) I0702 11:30:05.263343 1 tasks_processing.go:74] worker 1 stopped. E0702 11:30:05.263376 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0702 11:30:05.263692 1 recorder.go:75] Recording config/workload_info with fingerprint=c4849c84204edb962771e999c6d1fba2d5b6b14589ac28de5ce123f7c6e9fedb I0702 11:30:05.263708 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.349501006s to process 1 records E0702 11:30:05.263731 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.349s with: function \"workload_info\" failed with an error" I0702 11:30:05.264837 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0702 11:30:05.264850 1 periodic.go:209] Running conditional gatherer I0702 11:30:05.275888 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0702 11:30:05.282380 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.16:39285->172.30.0.10:53: read: connection refused E0702 11:30:05.282645 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0702 11:30:05.282711 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0702 11:30:05.289581 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0702 11:30:05.289595 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289600 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289603 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289606 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289610 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289613 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289616 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289618 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0702 11:30:05.289621 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0702 11:30:05.289634 1 tasks_processing.go:45] number of workers: 3 I0702 11:30:05.289642 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 11:30:05.289646 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0702 11:30:05.289650 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 11:30:05.289660 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0702 11:30:05.289665 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0702 11:30:05.289667 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 11:30:05.289676 1 tasks_processing.go:74] worker 1 stopped. I0702 11:30:05.289722 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0702 11:30:05.289736 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.235µs to process 1 records I0702 11:30:05.289803 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0702 11:30:05.289812 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 425ns to process 1 records I0702 11:30:05.289834 1 tasks_processing.go:74] worker 0 stopped. I0702 11:30:05.289907 1 tasks_processing.go:74] worker 2 stopped. I0702 11:30:05.289919 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 256.576µs to process 0 records I0702 11:30:05.289938 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.16:39285->172.30.0.10:53: read: connection refused I0702 11:30:05.289955 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0702 11:30:05.315031 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=d88ba47631b43e5766de1f4445e5b544391252bb88316a8f49391569d3b96a86 I0702 11:30:05.315155 1 diskrecorder.go:70] Writing 101 records to /var/lib/insights-operator/insights-2026-07-02-113005.tar.gz I0702 11:30:05.321704 1 diskrecorder.go:51] Wrote 101 records to disk in 6ms I0702 11:30:05.321739 1 periodic.go:278] Gathering cluster info every 2h0m0s I0702 11:30:05.321758 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0702 11:30:05.868883 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 11:30:06.067965 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 11:30:19.920560 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 11:31:03.625107 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="0bf6634dcb3417b4d61b8aa4b55e30f61ea6de3464239cd9a60388372696efa3") W0702 11:31:03.625139 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0702 11:31:03.625191 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="c27c07b53a35e5b3434f0f0a931b56fa2336d10579de7f336fea6807aaa3fd33") I0702 11:31:03.625207 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0702 11:31:03.625235 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0702 11:31:03.625278 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0702 11:31:03.625344 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting"