W0505 23:15:45.334738 1 cmd.go:257] Using insecure, self-signed certificates I0505 23:15:45.937392 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:15:45.937701 1 observer_polling.go:159] Starting file observer I0505 23:15:46.554543 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0505 23:15:46.554741 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0505 23:15:46.555319 1 secure_serving.go:57] Forcing use of http/1.1 only W0505 23:15:46.555339 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0505 23:15:46.555344 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0505 23:15:46.555349 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0505 23:15:46.555352 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. I0505 23:15:46.555347 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0505 23:15:46.555356 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0505 23:15:46.555375 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0505 23:15:46.559406 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0505 23:15:46.559431 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"bf3fe0fb-9398-4bc2-a58f-8b1a12e9309b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0505 23:15:46.561452 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0505 23:15:46.561476 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0505 23:15:46.561472 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0505 23:15:46.561476 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0505 23:15:46.561515 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0505 23:15:46.561519 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0505 23:15:46.561757 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1328885079/tls.crt::/tmp/serving-cert-1328885079/tls.key" I0505 23:15:46.562098 1 secure_serving.go:213] Serving securely on [::]:8443 I0505 23:15:46.562127 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0505 23:15:46.566168 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0505 23:15:46.566188 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0505 23:15:46.566304 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0505 23:15:46.572953 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:15:46.572978 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0505 23:15:46.580551 1 secretconfigobserver.go:119] support secret does not exist I0505 23:15:46.585655 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:15:46.594739 1 secretconfigobserver.go:119] support secret does not exist I0505 23:15:46.601888 1 recorder.go:161] Pruning old reports every 7h32m21s, max age is 288h0m0s I0505 23:15:46.607738 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0505 23:15:46.607750 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0505 23:15:46.607753 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0505 23:15:46.607760 1 insightsreport.go:296] Starting report retriever I0505 23:15:46.607770 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0505 23:15:46.607814 1 periodic.go:209] Running clusterconfig gatherer I0505 23:15:46.607869 1 tasks_processing.go:45] number of workers: 64 I0505 23:15:46.607899 1 tasks_processing.go:69] worker 3 listening for tasks. I0505 23:15:46.607907 1 tasks_processing.go:69] worker 19 listening for tasks. I0505 23:15:46.607911 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:15:46.607912 1 tasks_processing.go:71] worker 3 working on support_secret task. I0505 23:15:46.607932 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:15:46.607930 1 tasks_processing.go:71] worker 0 working on qemu_kubevirt_launcher_logs task. I0505 23:15:46.607934 1 tasks_processing.go:69] worker 35 listening for tasks. I0505 23:15:46.607937 1 tasks_processing.go:71] worker 1 working on machine_healthchecks task. I0505 23:15:46.607947 1 tasks_processing.go:69] worker 20 listening for tasks. I0505 23:15:46.607953 1 tasks_processing.go:69] worker 21 listening for tasks. I0505 23:15:46.607958 1 tasks_processing.go:69] worker 22 listening for tasks. I0505 23:15:46.607963 1 tasks_processing.go:69] worker 23 listening for tasks. I0505 23:15:46.607968 1 tasks_processing.go:69] worker 24 listening for tasks. I0505 23:15:46.607969 1 tasks_processing.go:69] worker 11 listening for tasks. I0505 23:15:46.607977 1 tasks_processing.go:69] worker 26 listening for tasks. I0505 23:15:46.607973 1 tasks_processing.go:69] worker 25 listening for tasks. I0505 23:15:46.607984 1 tasks_processing.go:69] worker 31 listening for tasks. I0505 23:15:46.607982 1 tasks_processing.go:69] worker 15 listening for tasks. I0505 23:15:46.607992 1 tasks_processing.go:69] worker 4 listening for tasks. I0505 23:15:46.607994 1 tasks_processing.go:69] worker 33 listening for tasks. I0505 23:15:46.607989 1 tasks_processing.go:69] worker 32 listening for tasks. I0505 23:15:46.608000 1 tasks_processing.go:69] worker 34 listening for tasks. I0505 23:15:46.608002 1 tasks_processing.go:69] worker 12 listening for tasks. I0505 23:15:46.607997 1 tasks_processing.go:69] worker 30 listening for tasks. I0505 23:15:46.608006 1 tasks_processing.go:69] worker 27 listening for tasks. I0505 23:15:46.608011 1 tasks_processing.go:69] worker 18 listening for tasks. I0505 23:15:46.608016 1 tasks_processing.go:69] worker 14 listening for tasks. I0505 23:15:46.608017 1 tasks_processing.go:69] worker 28 listening for tasks. I0505 23:15:46.608021 1 tasks_processing.go:71] worker 14 working on cost_management_metrics_configs task. I0505 23:15:46.608024 1 tasks_processing.go:69] worker 2 listening for tasks. I0505 23:15:46.608022 1 tasks_processing.go:69] worker 13 listening for tasks. I0505 23:15:46.608027 1 tasks_processing.go:69] worker 29 listening for tasks. I0505 23:15:46.608017 1 tasks_processing.go:71] worker 18 working on active_alerts task. I0505 23:15:46.608031 1 tasks_processing.go:69] worker 16 listening for tasks. I0505 23:15:46.608018 1 tasks_processing.go:69] worker 17 listening for tasks. I0505 23:15:46.608040 1 tasks_processing.go:71] worker 21 working on openstack_dataplanedeployments task. I0505 23:15:46.608031 1 tasks_processing.go:71] worker 35 working on tsdb_status task. I0505 23:15:46.608051 1 tasks_processing.go:69] worker 7 listening for tasks. I0505 23:15:46.608057 1 tasks_processing.go:71] worker 11 working on feature_gates task. I0505 23:15:46.608059 1 tasks_processing.go:69] worker 5 listening for tasks. I0505 23:15:46.608055 1 tasks_processing.go:69] worker 49 listening for tasks. W0505 23:15:46.608067 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:15:46.608072 1 tasks_processing.go:69] worker 9 listening for tasks. W0505 23:15:46.608074 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:15:46.608078 1 tasks_processing.go:69] worker 36 listening for tasks. I0505 23:15:46.608043 1 tasks_processing.go:69] worker 6 listening for tasks. I0505 23:15:46.608087 1 tasks_processing.go:71] worker 31 working on storage_classes task. I0505 23:15:46.608085 1 tasks_processing.go:69] worker 10 listening for tasks. I0505 23:15:46.608090 1 tasks_processing.go:71] worker 4 working on container_runtime_configs task. I0505 23:15:46.608096 1 tasks_processing.go:71] worker 12 working on crds task. I0505 23:15:46.608102 1 tasks_processing.go:69] worker 56 listening for tasks. I0505 23:15:46.608102 1 tasks_processing.go:69] worker 38 listening for tasks. I0505 23:15:46.608077 1 tasks_processing.go:71] worker 26 working on nodes task. I0505 23:15:46.608110 1 tasks_processing.go:69] worker 39 listening for tasks. I0505 23:15:46.608120 1 tasks_processing.go:69] worker 40 listening for tasks. I0505 23:15:46.608127 1 tasks_processing.go:69] worker 41 listening for tasks. I0505 23:15:46.608135 1 tasks_processing.go:69] worker 42 listening for tasks. I0505 23:15:46.608144 1 tasks_processing.go:69] worker 43 listening for tasks. I0505 23:15:46.608151 1 tasks_processing.go:69] worker 44 listening for tasks. I0505 23:15:46.608158 1 tasks_processing.go:69] worker 45 listening for tasks. I0505 23:15:46.608165 1 tasks_processing.go:69] worker 46 listening for tasks. I0505 23:15:46.608173 1 tasks_processing.go:69] worker 47 listening for tasks. I0505 23:15:46.608181 1 tasks_processing.go:69] worker 48 listening for tasks. I0505 23:15:46.608188 1 tasks_processing.go:71] worker 22 working on config_maps task. I0505 23:15:46.608261 1 tasks_processing.go:71] worker 24 working on ceph_cluster task. I0505 23:15:46.608327 1 tasks_processing.go:69] worker 57 listening for tasks. I0505 23:15:46.608036 1 tasks_processing.go:69] worker 8 listening for tasks. I0505 23:15:46.608036 1 tasks_processing.go:71] worker 20 working on infrastructures task. I0505 23:15:46.608351 1 tasks_processing.go:69] worker 58 listening for tasks. I0505 23:15:46.608356 1 tasks_processing.go:71] worker 2 working on aggregated_monitoring_cr_names task. I0505 23:15:46.608365 1 tasks_processing.go:69] worker 61 listening for tasks. I0505 23:15:46.608371 1 tasks_processing.go:71] worker 58 working on machine_autoscalers task. I0505 23:15:46.608376 1 tasks_processing.go:71] worker 61 working on validating_webhook_configurations task. I0505 23:15:46.607913 1 tasks_processing.go:71] worker 19 working on olm_operators task. I0505 23:15:46.608083 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 39.069µs to process 0 records I0505 23:15:46.608083 1 tasks_processing.go:71] worker 25 working on machine_configs task. I0505 23:15:46.608086 1 tasks_processing.go:71] worker 33 working on metrics task. I0505 23:15:46.608377 1 tasks_processing.go:69] worker 63 listening for tasks. W0505 23:15:46.608430 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:15:46.608449 1 tasks_processing.go:69] worker 60 listening for tasks. I0505 23:15:46.608366 1 tasks_processing.go:69] worker 62 listening for tasks. I0505 23:15:46.608458 1 tasks_processing.go:71] worker 17 working on jaegers task. I0505 23:15:46.608462 1 tasks_processing.go:69] worker 59 listening for tasks. I0505 23:15:46.608466 1 tasks_processing.go:71] worker 62 working on nodenetworkconfigurationpolicies task. I0505 23:15:46.608063 1 tasks_processing.go:71] worker 15 working on networks task. I0505 23:15:46.608478 1 tasks_processing.go:69] worker 50 listening for tasks. I0505 23:15:46.608485 1 tasks_processing.go:71] worker 50 working on machine_sets task. I0505 23:15:46.608508 1 tasks_processing.go:69] worker 53 listening for tasks. I0505 23:15:46.608519 1 tasks_processing.go:71] worker 53 working on container_images task. I0505 23:15:46.608547 1 tasks_processing.go:71] worker 60 working on oauths task. I0505 23:15:46.608563 1 tasks_processing.go:71] worker 6 working on ingress task. I0505 23:15:46.608451 1 tasks_processing.go:71] worker 13 working on service_accounts task. I0505 23:15:46.608574 1 tasks_processing.go:71] worker 38 working on image task. I0505 23:15:46.608438 1 tasks_processing.go:71] worker 28 working on image_registries task. I0505 23:15:46.608448 1 tasks_processing.go:71] worker 56 working on certificate_signing_requests task. I0505 23:15:46.608468 1 tasks_processing.go:71] worker 59 working on openstack_controlplanes task. I0505 23:15:46.608095 1 tasks_processing.go:69] worker 37 listening for tasks. I0505 23:15:46.608456 1 tasks_processing.go:71] worker 16 working on mutating_webhook_configurations task. I0505 23:15:46.608091 1 tasks_processing.go:71] worker 30 working on openshift_machine_api_events task. I0505 23:15:46.608607 1 tasks_processing.go:71] worker 45 working on ingress_certificates task. I0505 23:15:46.608618 1 tasks_processing.go:71] worker 46 working on version task. I0505 23:15:46.608619 1 tasks_processing.go:71] worker 48 working on machine_config_pools task. I0505 23:15:46.608454 1 tasks_processing.go:71] worker 29 working on clusterroles task. I0505 23:15:46.608642 1 tasks_processing.go:71] worker 37 working on openshift_logging task. I0505 23:15:46.608603 1 tasks_processing.go:71] worker 47 working on monitoring_persistent_volumes task. I0505 23:15:46.608722 1 tasks_processing.go:71] worker 35 working on install_plans task. I0505 23:15:46.608959 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 31.081µs to process 0 records I0505 23:15:46.608977 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 41.957µs to process 0 records I0505 23:15:46.609334 1 tasks_processing.go:69] worker 54 listening for tasks. I0505 23:15:46.609345 1 tasks_processing.go:71] worker 54 working on cluster_apiserver task. I0505 23:15:46.609599 1 tasks_processing.go:71] worker 33 working on storage_cluster task. I0505 23:15:46.609992 1 tasks_processing.go:69] worker 52 listening for tasks. I0505 23:15:46.610008 1 tasks_processing.go:71] worker 52 working on image_pruners task. I0505 23:15:46.610199 1 tasks_processing.go:69] worker 51 listening for tasks. I0505 23:15:46.608519 1 tasks_processing.go:71] worker 63 working on overlapping_namespace_uids task. I0505 23:15:46.610296 1 tasks_processing.go:74] worker 51 stopped. I0505 23:15:46.610415 1 tasks_processing.go:71] worker 32 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0505 23:15:46.608526 1 tasks_processing.go:71] worker 18 working on openstack_dataplanenodesets task. I0505 23:15:46.608541 1 tasks_processing.go:71] worker 49 working on authentication task. I0505 23:15:46.610841 1 tasks_processing.go:71] worker 36 working on sap_datahubs task. I0505 23:15:46.608532 1 tasks_processing.go:71] worker 7 working on operators_pods_and_events task. I0505 23:15:46.608567 1 tasks_processing.go:71] worker 10 working on lokistack task. I0505 23:15:46.608536 1 tasks_processing.go:71] worker 5 working on openstack_version task. I0505 23:15:46.611011 1 tasks_processing.go:71] worker 40 working on machines task. I0505 23:15:46.608613 1 tasks_processing.go:71] worker 57 working on operators task. I0505 23:15:46.611016 1 tasks_processing.go:71] worker 34 working on pdbs task. I0505 23:15:46.608561 1 tasks_processing.go:71] worker 9 working on silenced_alerts task. I0505 23:15:46.611025 1 tasks_processing.go:71] worker 8 working on proxies task. I0505 23:15:46.611057 1 tasks_processing.go:71] worker 39 working on sap_pods task. I0505 23:15:46.611360 1 tasks_processing.go:71] worker 44 working on schedulers task. I0505 23:15:46.611393 1 tasks_processing.go:69] worker 55 listening for tasks. I0505 23:15:46.611752 1 tasks_processing.go:74] worker 55 stopped. I0505 23:15:46.608600 1 tasks_processing.go:71] worker 42 working on dvo_metrics task. I0505 23:15:46.608607 1 tasks_processing.go:71] worker 43 working on node_logs task. I0505 23:15:46.611972 1 tasks_processing.go:71] worker 27 working on pod_network_connectivity_checks task. I0505 23:15:46.612073 1 tasks_processing.go:71] worker 41 working on nodenetworkstates task. I0505 23:15:46.612354 1 tasks_processing.go:71] worker 23 working on sap_config task. W0505 23:15:46.612559 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:15:46.612579 1 tasks_processing.go:74] worker 9 stopped. I0505 23:15:46.612590 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.426119ms to process 0 records I0505 23:15:46.616969 1 tasks_processing.go:74] worker 17 stopped. I0505 23:15:46.616989 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 8.493098ms to process 0 records I0505 23:15:46.617001 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 8.922404ms to process 0 records I0505 23:15:46.617011 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 8.944334ms to process 0 records I0505 23:15:46.617019 1 tasks_processing.go:74] worker 14 stopped. I0505 23:15:46.617024 1 tasks_processing.go:74] worker 21 stopped. I0505 23:15:46.617162 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0505 23:15:46.617181 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0505 23:15:46.617186 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0505 23:15:46.617190 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0505 23:15:46.617205 1 controller.go:489] The operator is still being initialized I0505 23:15:46.617212 1 controller.go:512] The operator is healthy I0505 23:15:46.630561 1 tasks_processing.go:74] worker 3 stopped. E0505 23:15:46.630582 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0505 23:15:46.630592 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 22.620154ms to process 0 records I0505 23:15:46.630630 1 gather_logs.go:145] no pods in namespace were found I0505 23:15:46.630643 1 tasks_processing.go:74] worker 0 stopped. I0505 23:15:46.630667 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 22.700928ms to process 0 records I0505 23:15:46.630913 1 tasks_processing.go:74] worker 11 stopped. I0505 23:15:46.631136 1 recorder.go:75] Recording config/featuregate with fingerprint=017946c011b49be3fbde1da526ba2471c2f11cf01415f18aa362d1c639a38e8b I0505 23:15:46.631154 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 22.841569ms to process 1 records I0505 23:15:46.644742 1 tasks_processing.go:74] worker 4 stopped. I0505 23:15:46.644761 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 36.63496ms to process 0 records I0505 23:15:46.644768 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 36.462148ms to process 0 records I0505 23:15:46.644772 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 36.384218ms to process 0 records I0505 23:15:46.644775 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 35.151627ms to process 0 records I0505 23:15:46.644780 1 tasks_processing.go:74] worker 33 stopped. I0505 23:15:46.644779 1 tasks_processing.go:74] worker 24 stopped. I0505 23:15:46.644785 1 tasks_processing.go:74] worker 58 stopped. I0505 23:15:46.644907 1 tasks_processing.go:74] worker 1 stopped. E0505 23:15:46.644937 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0505 23:15:46.644956 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 36.956463ms to process 0 records I0505 23:15:46.645238 1 tasks_processing.go:74] worker 6 stopped. I0505 23:15:46.645445 1 recorder.go:75] Recording config/ingress with fingerprint=62a78dbc9d06bf0b8b3f4c06c8043930712fc42a9c5ef002e0c7473d9e242589 I0505 23:15:46.645460 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 36.662381ms to process 1 records I0505 23:15:46.645552 1 tasks_processing.go:74] worker 52 stopped. I0505 23:15:46.645657 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=47e49caa6916dd280fe672e58782dae2287c1865a60b0b809ea69eba76e37724 I0505 23:15:46.645667 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 35.410587ms to process 1 records I0505 23:15:46.661166 1 tasks_processing.go:74] worker 62 stopped. I0505 23:15:46.661187 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 52.67453ms to process 0 records I0505 23:15:46.661302 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=f4e4e3f95fe90981d9d0e8285131c6ad0c437c00c45f27e5f817b99f72fb05ce I0505 23:15:46.661322 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=d468dff99c2d2abb0a3d9ba5c436088dee98e7ad9bff1a6ca29072d6c4e964b8 I0505 23:15:46.661331 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 53.093046ms to process 2 records I0505 23:15:46.661342 1 tasks_processing.go:74] worker 31 stopped. E0505 23:15:46.661351 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0505 23:15:46.661360 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 49.227458ms to process 0 records I0505 23:15:46.661367 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 48.894995ms to process 0 records I0505 23:15:46.661374 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 50.764544ms to process 0 records I0505 23:15:46.661380 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 50.210395ms to process 0 records I0505 23:15:46.661386 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 49.298815ms to process 0 records I0505 23:15:46.661391 1 tasks_processing.go:74] worker 41 stopped. I0505 23:15:46.661397 1 tasks_processing.go:74] worker 27 stopped. I0505 23:15:46.661398 1 tasks_processing.go:74] worker 18 stopped. I0505 23:15:46.661408 1 tasks_processing.go:74] worker 5 stopped. I0505 23:15:46.661409 1 tasks_processing.go:74] worker 23 stopped. I0505 23:15:46.661429 1 tasks_processing.go:74] worker 59 stopped. I0505 23:15:46.661443 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 52.828233ms to process 0 records I0505 23:15:46.661458 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 50.561351ms to process 0 records I0505 23:15:46.661465 1 tasks_processing.go:74] worker 36 stopped. I0505 23:15:46.661469 1 tasks_processing.go:74] worker 38 stopped. I0505 23:15:46.661512 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0505 23:15:46.661550 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0505 23:15:46.661550 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0505 23:15:46.661579 1 recorder.go:75] Recording config/image with fingerprint=0ab3de24c91ecfc10540d37fb70d3503527009017eeb3874e7e7a1a451f32741 I0505 23:15:46.661589 1 gather.go:177] gatherer "clusterconfig" function "image" took 52.884963ms to process 1 records I0505 23:15:46.661695 1 tasks_processing.go:74] worker 20 stopped. I0505 23:15:46.662196 1 recorder.go:75] Recording config/infrastructure with fingerprint=5e67aefbd5c962f009d325789840560f4debd017e4e8d2045e9069344549d93a I0505 23:15:46.662211 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 53.120186ms to process 1 records I0505 23:15:46.662221 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 53.041415ms to process 0 records I0505 23:15:46.662236 1 tasks_processing.go:74] worker 50 stopped. I0505 23:15:46.662298 1 tasks_processing.go:74] worker 49 stopped. I0505 23:15:46.663334 1 recorder.go:75] Recording config/authentication with fingerprint=0e32006543d7769fafaced795779d5e73847ecc2381c9ca3c314cd93c2638816 I0505 23:15:46.663363 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0505 23:15:46.663369 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 50.837106ms to process 1 records I0505 23:15:46.663521 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0505 23:15:46.663957 1 tasks_processing.go:74] worker 15 stopped. I0505 23:15:46.663977 1 recorder.go:75] Recording config/network with fingerprint=e415d8863e9cffa191cf928d8a3e8838fc6f4e53c0bf198b6fb3c9fec610cea7 I0505 23:15:46.664056 1 gather.go:177] gatherer "clusterconfig" function "networks" took 53.174017ms to process 1 records I0505 23:15:46.664157 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 53.101834ms to process 0 records W0505 23:15:46.664167 1 operator.go:288] started E0505 23:15:46.664172 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0505 23:15:46.664181 1 gather.go:177] gatherer "clusterconfig" function "machines" took 50.649894ms to process 0 records I0505 23:15:46.664191 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 53.242546ms to process 0 records I0505 23:15:46.664204 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 53.48028ms to process 0 records I0505 23:15:46.664786 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0505 23:15:46.664807 1 tasks_processing.go:74] worker 60 stopped. I0505 23:15:46.664813 1 tasks_processing.go:74] worker 37 stopped. I0505 23:15:46.664819 1 tasks_processing.go:74] worker 40 stopped. I0505 23:15:46.664822 1 tasks_processing.go:74] worker 48 stopped. I0505 23:15:46.664825 1 tasks_processing.go:74] worker 30 stopped. I0505 23:15:46.665017 1 recorder.go:75] Recording config/oauth with fingerprint=c2d81a8d29b6a6d7ccda39a627cc13c40d222704795725a99031aca15b11fd77 I0505 23:15:46.665036 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 53.551929ms to process 1 records I0505 23:15:46.665119 1 tasks_processing.go:74] worker 54 stopped. I0505 23:15:46.665161 1 recorder.go:75] Recording config/apiserver with fingerprint=e9ab684bc37b16947e791211aabe655f152c692a18bbeb9603d180f963066fbe I0505 23:15:46.665173 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 53.149779ms to process 1 records I0505 23:15:46.665246 1 tasks_processing.go:74] worker 26 stopped. I0505 23:15:46.665461 1 recorder.go:75] Recording config/node/ip-10-0-2-80.ec2.internal with fingerprint=b250015a325ee087577b9385b4223ed4f1af6acbca766078fce0c738dcdd1082 I0505 23:15:46.665478 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 54.579768ms to process 1 records I0505 23:15:46.665486 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 51.895956ms to process 0 records I0505 23:15:46.665491 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 51.864473ms to process 0 records I0505 23:15:46.665544 1 tasks_processing.go:74] worker 39 stopped. I0505 23:15:46.665547 1 tasks_processing.go:74] worker 10 stopped. I0505 23:15:46.665613 1 tasks_processing.go:74] worker 61 stopped. I0505 23:15:46.665699 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=f4e429ca181b2e0d80f69c987d53470eb4f8ae97d63bb9c57159e370de7b09b9 I0505 23:15:46.665788 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=bc641c730110cd9fb1060b6b28895bb4fe2ba38ef27b9c869be36a2adf7e9ab2 I0505 23:15:46.665809 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=60551444c8f81eb6d41809d30f5988ca57de59cb31714c416c708d964ab9f36d I0505 23:15:46.665836 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=32436e41b748c85ac67c634eb991f349e5b08665304e337367294dcabf21759d I0505 23:15:46.665861 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=8b5c3beff2453722abaae8d4e5ec6c4d7240a6c9a5f453f89efae34c24307d4e I0505 23:15:46.665888 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=8d8f64b069aaa219edec2c73fdf8dd40d7a632016db8e110862c856e89b42fb3 I0505 23:15:46.665939 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=e984353d538022aaa5164e466af0e7dcf9fedb2ae93235b19c23b3a3cabe144b I0505 23:15:46.665992 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=29c89a009f54b7a94e0143a84a436e90f1b20cedb8c7cb43a32df9a21ae5a617 I0505 23:15:46.666029 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=70975b708d426fe305ca512d1bc0d0b71111a21d6ad1467a1d8bfcffef1033df I0505 23:15:46.666058 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=6a6a3b37bc9a4e1dfaa5f134f1f631d139052a0183a2858e2f7e76e510c3a2d2 I0505 23:15:46.666098 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=5dce82befd2c6f3e76ced1a9cc748c1c097fcc7f1760febcd33debc7c66981f4 I0505 23:15:46.666111 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 55.169404ms to process 11 records I0505 23:15:46.666171 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=3ae80cd681d63df294b9e282175320cdca6bbb2d89dc6e9e9e7ecf7a7f30b76f I0505 23:15:46.666181 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 52.404352ms to process 1 records I0505 23:15:46.666193 1 tasks_processing.go:74] worker 44 stopped. I0505 23:15:46.666264 1 tasks_processing.go:74] worker 28 stopped. I0505 23:15:46.666431 1 base_controller.go:82] Caches are synced for ConfigController I0505 23:15:46.666442 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0505 23:15:46.666604 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f32b966ef6abe9096058ba406fd01a0eece6f884e0f11114c47bc06315c54b09 I0505 23:15:46.666623 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 56.380572ms to process 1 records I0505 23:15:46.666634 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 56.669615ms to process 0 records I0505 23:15:46.666640 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 56.64557ms to process 0 records I0505 23:15:46.666700 1 recorder.go:75] Recording config/proxy with fingerprint=dd7d16adea67f59e708c7a6d1bbbf8dd3e64be5871d8aac6670a223c48de8ac4 I0505 23:15:46.666702 1 tasks_processing.go:74] worker 19 stopped. I0505 23:15:46.666712 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 54.189398ms to process 1 records I0505 23:15:46.666717 1 tasks_processing.go:74] worker 47 stopped. I0505 23:15:46.666718 1 tasks_processing.go:74] worker 8 stopped. I0505 23:15:46.666741 1 tasks_processing.go:74] worker 56 stopped. I0505 23:15:46.666756 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 58.140229ms to process 0 records I0505 23:15:46.667649 1 tasks_processing.go:74] worker 63 stopped. I0505 23:15:46.667678 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0505 23:15:46.667689 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 57.363095ms to process 1 records I0505 23:15:46.668165 1 tasks_processing.go:74] worker 34 stopped. I0505 23:15:46.668293 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=7939c82bac0af757892656fec33b45912225c8ba621b50138f8faa28450fba67 I0505 23:15:46.668331 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=80a548ced30e132c094478aafabb65758dd12e8e5b82a54a43a1cc351e493f9f I0505 23:15:46.668368 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=b3916c5b83778ff731710289ddb42d973d27798c1ac4f21e3b451b5f923a58f7 I0505 23:15:46.668382 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 57.007796ms to process 3 records W0505 23:15:46.671023 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0505 23:15:46.671065 1 tasks_processing.go:74] worker 16 stopped. I0505 23:15:46.671223 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=6688d910881cd30c3dcec51fe8087a7ab8aab92f557539b702c52f7b8a36b9ef I0505 23:15:46.671275 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=b738d3888aeb0c3158e271d92603435bbd69cfce2b88270fb67f16f5e04761f0 I0505 23:15:46.671318 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=59b706fef2ac15d8443e6bd7fb740ca83e974b429e599a219ab48a416c988083 I0505 23:15:46.671333 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 62.452744ms to process 3 records I0505 23:15:46.672663 1 tasks_processing.go:74] worker 12 stopped. I0505 23:15:46.673185 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=c3c81e1abb1ad0869c3be99cd383d37d0d4dd237ff0229fd757b35efb4800fed I0505 23:15:46.673377 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=f526881972de70ad4c21066dd27f545a13b763402dfe017b3ef7f4168f72356a I0505 23:15:46.673386 1 gather.go:177] gatherer "clusterconfig" function "crds" took 64.55656ms to process 2 records I0505 23:15:46.677247 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:15:46.677476 1 tasks_processing.go:74] worker 46 stopped. I0505 23:15:46.677632 1 controller.go:212] Source scaController *sca.Controller is not ready I0505 23:15:46.677643 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0505 23:15:46.677647 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0505 23:15:46.677650 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0505 23:15:46.677652 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0505 23:15:46.677668 1 controller.go:489] The operator is still being initialized I0505 23:15:46.677676 1 controller.go:512] The operator is healthy I0505 23:15:46.677724 1 recorder.go:75] Recording config/version with fingerprint=9d01b4922bd3e6235bbd8574d9b557194d4b742457456e0b8cc9438b2107f0e7 I0505 23:15:46.677736 1 recorder.go:75] Recording config/id with fingerprint=649c4230ff734cf2a692119fb7628411b3f77a84576d71a4270584aba1b1a4fd I0505 23:15:46.677743 1 gather.go:177] gatherer "clusterconfig" function "version" took 68.848312ms to process 2 records I0505 23:15:46.681218 1 tasks_processing.go:74] worker 32 stopped. I0505 23:15:46.681238 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 70.781132ms to process 0 records I0505 23:15:46.681796 1 prometheus_rules.go:88] Prometheus rules successfully created I0505 23:15:46.683250 1 tasks_processing.go:74] worker 29 stopped. I0505 23:15:46.683881 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=ba291e4f7f820121562aed2da85738b325d4d35abec67a2ecd20eb47d07176f0 I0505 23:15:46.684012 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=a398cb229bf0efd84062bbe738525590b687eb826d2e700a2db2562b7274e122 I0505 23:15:46.684024 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 74.600427ms to process 2 records I0505 23:15:46.684295 1 tasks_processing.go:74] worker 2 stopped. I0505 23:15:46.684319 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 75.920459ms to process 0 records I0505 23:15:46.685222 1 tasks_processing.go:74] worker 53 stopped. E0505 23:15:46.686191 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%270d5cc4c0-9443-4501-8894-9baa96e61e08%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:36821->172.30.0.10:53: read: connection refused I0505 23:15:46.686209 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%270d5cc4c0-9443-4501-8894-9baa96e61e08%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:36821->172.30.0.10:53: read: connection refused I0505 23:15:46.686691 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-gqsh9 with fingerprint=72ba20b8613778d156eac46d5b24f48a5bd59e02125b89944eb26896fa72abea I0505 23:15:46.686756 1 recorder.go:75] Recording config/running_containers with fingerprint=6271cd255a464c3b6f89628d0295519aed28c8d54ae0ee9ae7d1ec81e5bfe05c I0505 23:15:46.686766 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 76.687602ms to process 2 records I0505 23:15:46.705755 1 tasks_processing.go:74] worker 22 stopped. E0505 23:15:46.705776 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0505 23:15:46.705782 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0505 23:15:46.705786 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0505 23:15:46.705798 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0505 23:15:46.705830 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0505 23:15:46.705838 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0505 23:15:46.705843 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0505 23:15:46.705847 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0505 23:15:46.705885 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0505 23:15:46.705894 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0505 23:15:46.705899 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 97.549774ms to process 7 records I0505 23:15:46.707363 1 tasks_processing.go:74] worker 45 stopped. E0505 23:15:46.707381 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0505 23:15:46.707387 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q42g1qv9b034r6e5947ge0p9cjkt03e-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q42g1qv9b034r6e5947ge0p9cjkt03e-primary-cert-bundle-secret" not found I0505 23:15:46.707439 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=3c0b0469d3f86357d624cf9db427ef52cd99192e14732fb76583d3e51118b322 I0505 23:15:46.707453 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 98.738652ms to process 1 records I0505 23:15:46.746210 1 tasks_processing.go:74] worker 43 stopped. I0505 23:15:46.746231 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 134.297128ms to process 0 records I0505 23:15:46.750019 1 tasks_processing.go:74] worker 25 stopped. I0505 23:15:46.750052 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0505 23:15:46.750064 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 141.599755ms to process 1 records I0505 23:15:46.758710 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0505 23:15:46.762451 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:36299->172.30.0.10:53: read: connection refused I0505 23:15:46.762466 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:36299->172.30.0.10:53: read: connection refused I0505 23:15:46.765544 1 base_controller.go:82] Caches are synced for LoggingSyncer I0505 23:15:46.765557 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0505 23:15:47.081134 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0505 23:15:47.081154 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0505 23:15:47.081193 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-9hzcw pod in namespace openshift-dns (previous: false). I0505 23:15:47.305300 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-9hzcw pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-9hzcw\" is waiting to start: ContainerCreating" I0505 23:15:47.305320 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-9hzcw\" is waiting to start: ContainerCreating" I0505 23:15:47.305329 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-9hzcw pod in namespace openshift-dns (previous: false). I0505 23:15:47.483861 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-9hzcw pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-9hzcw\" is waiting to start: ContainerCreating" I0505 23:15:47.483881 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-9hzcw\" is waiting to start: ContainerCreating" I0505 23:15:47.483892 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-lpwvw pod in namespace openshift-dns (previous: false). W0505 23:15:47.667018 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0505 23:15:47.684313 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0505 23:15:47.684353 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-77c986f646-p2sgv pod in namespace openshift-image-registry (previous: false). I0505 23:15:47.878123 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0505 23:15:47.878142 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-78f7d69757-7jqnd pod in namespace openshift-image-registry (previous: false). I0505 23:15:48.079556 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0505 23:15:48.079577 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-78f7d69757-x5bzc pod in namespace openshift-image-registry (previous: false). I0505 23:15:48.282744 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-78f7d69757-x5bzc pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-78f7d69757-x5bzc\" is waiting to start: ContainerCreating" I0505 23:15:48.282762 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-78f7d69757-x5bzc\" is waiting to start: ContainerCreating" I0505 23:15:48.282773 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-762bp pod in namespace openshift-image-registry (previous: false). I0505 23:15:48.292342 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0505 23:15:48.485478 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0505 23:15:48.485501 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-76f9f7768-c6mqf pod in namespace openshift-ingress (previous: false). W0505 23:15:48.667715 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0505 23:15:48.680295 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0505 23:15:48.680316 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-76f9f7768-pj2mp pod in namespace openshift-ingress (previous: false). I0505 23:15:48.882743 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-76f9f7768-pj2mp pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-76f9f7768-pj2mp\" is waiting to start: ContainerCreating" I0505 23:15:48.882760 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-76f9f7768-pj2mp\" is waiting to start: ContainerCreating" I0505 23:15:48.882772 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-99798445c-x2wqh pod in namespace openshift-ingress (previous: false). I0505 23:15:48.894482 1 tasks_processing.go:74] worker 57 stopped. I0505 23:15:48.894532 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=5bfacf0b8c0f0664ffd068e12ba8b274a33295f766e66ae840d2506ec38d32b2 I0505 23:15:48.894564 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=9225734c487f33b0b88072dfe37b46a2c692c0e55cb86582931320c5d16266f1 I0505 23:15:48.894592 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0505 23:15:48.894620 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=cfdde0f0a3853d2b785a7f5a31caa45132994ee2672ee4df19da5b2824b5e70a I0505 23:15:48.894637 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0505 23:15:48.894658 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=f40cbb8aa0333e6d359e05511824fe325fe80d58fe0939bbd649a4e6da4abd31 I0505 23:15:48.894689 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=f56c38d4a88cba2afced1457c62ef3fbfe4ea1fb17ad8a9e0334ffae35c41abb I0505 23:15:48.894728 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=81cc14e824e7a8ed0059e42c82078007d9c0f37e80f3914d4332d69cc179ae50 I0505 23:15:48.894761 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=8d3bc6ca98e073563b835be029dfc71a3d275f3f5992529d7d9a1a4adde96eb3 I0505 23:15:48.894770 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0505 23:15:48.894787 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=79ee360b378a7e4c43538f3cff2400790768522a4980c17cfd39739796861acf I0505 23:15:48.894795 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0505 23:15:48.894812 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=02c9b294508cae082e1b1992256fed3b4051382fc874014d1ab1e96be9f74f22 I0505 23:15:48.894823 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0505 23:15:48.894838 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=c7d5fe43cf6e919060357ea2be0902247db2dbf189843e132a3d796ddb33be47 I0505 23:15:48.894845 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0505 23:15:48.894859 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c743aa127ac1af82a1c729393da5b3027bb26d1b43aa06e9cd666d63b1f91241 I0505 23:15:48.894866 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0505 23:15:48.894881 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=8ba1204595950aab56673d5b7ffadf00a8738997c3f1c8472b995f8eccd836a1 I0505 23:15:48.895039 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=c1ac9807ac35645cf26c36fefdff0eca6518d4e5b1e3f463eed01658fba5e636 I0505 23:15:48.895052 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0505 23:15:48.895061 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0505 23:15:48.895081 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0505 23:15:48.895108 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=2ae87a25f7855eb21de8206826e9738c01edd39225b9a4c28b0c2cf1037b77ed I0505 23:15:48.895137 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=b65a4a2162f7face1ddb0bca0131c532e781923df46f3e484c2fd36f6363682c I0505 23:15:48.895146 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0505 23:15:48.895163 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=577050bf201416177211a2e699ce527719596c1b410840d7ebf947af33086be6 I0505 23:15:48.895173 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0505 23:15:48.895195 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=4f1b1d3a92ec1ba02dc2321de887e329908f6408b6d2097a2d4300b123035031 I0505 23:15:48.895211 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=7908415c9dfb00ae280db37e6dc2e74dc27ae8b9fec9642b3562e2549acbc86b I0505 23:15:48.895225 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=7f4aaa10a8ae77444facefade9f3017696c55f37f5b1d8eca03e23a10c833022 I0505 23:15:48.895239 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=1c882f5d54b8447e70d8c6057621704be051f083e0c08e8c91004916f46bdc5d I0505 23:15:48.895262 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=8f5178bd6fd21ae4a0469b69bc8212d5c558ee4d48cc7944d684936c375f25c7 I0505 23:15:48.895274 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0505 23:15:48.895301 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=7565aef245637597a9943b31162016b3be18b2795090c62d0526f1b093ad404e I0505 23:15:48.895319 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0505 23:15:48.895325 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0505 23:15:48.895332 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.283327929s to process 37 records I0505 23:15:49.083444 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-99798445c-x2wqh pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-99798445c-x2wqh\" is waiting to start: ContainerCreating" I0505 23:15:49.083467 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-99798445c-x2wqh\" is waiting to start: ContainerCreating" I0505 23:15:49.083483 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-hmfrm pod in namespace openshift-ingress-canary (previous: false). I0505 23:15:49.286739 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-hmfrm pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-hmfrm\" is waiting to start: ContainerCreating" I0505 23:15:49.286764 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-hmfrm\" is waiting to start: ContainerCreating" I0505 23:15:49.286786 1 tasks_processing.go:74] worker 7 stopped. I0505 23:15:49.286909 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=25965d0511c0f5352b9d2ed3d506bc46f1cc29b6bf2a3a9434f720a8e8c39916 I0505 23:15:49.286976 1 recorder.go:75] Recording events/openshift-dns with fingerprint=6ae9c2abbb67bc1cba7f1f424e0bbc055e6ae36af1e8d96b3f51a29beaff03da I0505 23:15:49.287059 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=ded9a5a0642de76ea3a021b2a2266b5cd0f3cf30b0f2f7fe8a4a2e8b010b6d7c I0505 23:15:49.287100 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=4cd635371359c44f0876af318f39c5119910f088ee356f8e265d37aa1f3a0797 I0505 23:15:49.287159 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=4f45d533e020ac35e54b931effc55b1be73795606314736e984102daafcdddfd I0505 23:15:49.287178 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=2a1ddf18f4280764c1c857d1488e0135e4a99b0e07cb0ad2370b159d0e552643 I0505 23:15:49.287190 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.675805552s to process 6 records W0505 23:15:49.667533 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0505 23:15:50.666952 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0505 23:15:51.666990 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0505 23:15:51.667019 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0505 23:15:51.667035 1 tasks_processing.go:74] worker 42 stopped. E0505 23:15:51.667045 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0505 23:15:51.667057 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0505 23:15:51.667071 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0505 23:15:51.667081 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.055173384s to process 1 records I0505 23:15:57.716558 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:15:57.919759 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:15:58.613623 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:15:59.276379 1 tasks_processing.go:74] worker 35 stopped. I0505 23:15:59.276426 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0505 23:15:59.276440 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.667633066s to process 1 records I0505 23:16:00.019637 1 tasks_processing.go:74] worker 13 stopped. I0505 23:16:00.019940 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=e07c889be8bf997ab16db7302dcb8e4965e61a90db1a5dd1c22a75d836224b34 I0505 23:16:00.019957 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.411047823s to process 1 records E0505 23:16:00.020014 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.412s with: function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0505 23:16:00.021132 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0505 23:16:00.021148 1 periodic.go:209] Running workloads gatherer I0505 23:16:00.021167 1 tasks_processing.go:45] number of workers: 2 I0505 23:16:00.021177 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:16:00.021181 1 tasks_processing.go:71] worker 1 working on workload_info task. I0505 23:16:00.021196 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:16:00.021208 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0505 23:16:00.039748 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0505 23:16:00.046961 1 tasks_processing.go:74] worker 0 stopped. I0505 23:16:00.046981 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.732013ms to process 0 records I0505 23:16:00.048221 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (9ms) I0505 23:16:00.056313 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (8ms) I0505 23:16:00.064371 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (8ms) I0505 23:16:00.072212 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0505 23:16:00.080178 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (8ms) I0505 23:16:00.089550 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (9ms) I0505 23:16:00.099641 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (10ms) I0505 23:16:00.108034 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (8ms) I0505 23:16:00.117762 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (10ms) I0505 23:16:00.128240 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (10ms) I0505 23:16:00.150254 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (22ms) I0505 23:16:00.250174 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (100ms) I0505 23:16:00.350508 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0505 23:16:00.449595 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (99ms) I0505 23:16:00.548608 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0505 23:16:00.649064 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0505 23:16:00.748875 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0505 23:16:00.848268 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (99ms) I0505 23:16:00.947943 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0505 23:16:01.048757 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (101ms) I0505 23:16:01.149121 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0505 23:16:01.249046 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0505 23:16:01.348754 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0505 23:16:01.448400 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0505 23:16:01.550808 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (102ms) I0505 23:16:01.649209 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (98ms) I0505 23:16:01.750160 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0505 23:16:01.849163 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (99ms) I0505 23:16:01.949672 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0505 23:16:02.048885 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (99ms) I0505 23:16:02.148896 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0505 23:16:02.148954 1 tasks_processing.go:74] worker 1 stopped. E0505 23:16:02.148969 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0505 23:16:02.149280 1 recorder.go:75] Recording config/workload_info with fingerprint=33b569d26030644511f75db137d3525ef471e641a8b9f6be8810dfe6173942dc I0505 23:16:02.149301 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.127764252s to process 1 records E0505 23:16:02.149335 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.128s with: function \"workload_info\" failed with an error" I0505 23:16:02.150448 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0505 23:16:02.150466 1 periodic.go:209] Running conditional gatherer I0505 23:16:02.157127 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0505 23:16:02.165927 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:54965->172.30.0.10:53: read: connection refused E0505 23:16:02.166212 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0505 23:16:02.166278 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0505 23:16:02.171935 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0505 23:16:02.171954 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171961 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171965 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171968 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171971 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171974 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171977 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171980 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0505 23:16:02.171983 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0505 23:16:02.172003 1 tasks_processing.go:45] number of workers: 3 I0505 23:16:02.172021 1 tasks_processing.go:69] worker 2 listening for tasks. I0505 23:16:02.172030 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0505 23:16:02.172030 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:16:02.172044 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:16:02.172062 1 tasks_processing.go:74] worker 1 stopped. I0505 23:16:02.172051 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0505 23:16:02.172062 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0505 23:16:02.172148 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0505 23:16:02.172163 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.372µs to process 1 records I0505 23:16:02.172203 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0505 23:16:02.172211 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.852µs to process 1 records I0505 23:16:02.172217 1 tasks_processing.go:74] worker 0 stopped. I0505 23:16:02.172326 1 tasks_processing.go:74] worker 2 stopped. I0505 23:16:02.172343 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 249.44µs to process 0 records I0505 23:16:02.172372 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:54965->172.30.0.10:53: read: connection refused I0505 23:16:02.172390 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0505 23:16:02.194348 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=fff00a043fcb1b0326fd373b9e754ebbdfa287e4eb58a0e05e798dc24ffa56cc I0505 23:16:02.194490 1 diskrecorder.go:70] Writing 100 records to /var/lib/insights-operator/insights-2026-05-05-231602.tar.gz I0505 23:16:02.200411 1 diskrecorder.go:51] Wrote 100 records to disk in 6ms I0505 23:16:02.200454 1 periodic.go:278] Gathering cluster info every 2h0m0s I0505 23:16:02.200473 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0505 23:16:11.226345 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:17:10.938587 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="a45e5f5b0e834b74a183beeb38bb2a4094553db829e76636afee391b3afe5f49") W0505 23:17:10.938618 1 builder.go:160] Restart triggered because of file /var/run/configmaps/service-ca-bundle/service-ca.crt was created I0505 23:17:10.938671 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0505 23:17:10.938682 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="a840dace3c8e519e058f95ecb9d97aaf094d0085f1fc9335e5f0b590d0c64f6d") I0505 23:17:10.938719 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="3fbd4602c2a02570cf3d2f312c10e1de0b79efcfccbd4cd03d49fd45f57be167") I0505 23:17:10.938719 1 base_controller.go:181] Shutting down ConfigController ... I0505 23:17:10.938732 1 base_controller.go:181] Shutting down LoggingSyncer ... I0505 23:17:10.938760 1 periodic.go:170] Shutting down I0505 23:17:10.938776 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0505 23:17:10.938785 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0505 23:17:10.938783 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped"