W0507 11:21:36.919043 1 cmd.go:257] Using insecure, self-signed certificates I0507 11:21:37.635890 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:37.636198 1 observer_polling.go:159] Starting file observer I0507 11:21:37.997624 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0507 11:21:37.997838 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0507 11:21:37.998450 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0507 11:21:37.998965 1 secure_serving.go:57] Forcing use of http/1.1 only W0507 11:21:37.998981 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0507 11:21:37.998986 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0507 11:21:37.998991 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0507 11:21:37.998993 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0507 11:21:37.998996 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0507 11:21:37.998998 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0507 11:21:38.002914 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0507 11:21:38.002954 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"16722342-b794-4630-b270-29d240195b76", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0507 11:21:38.003988 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0507 11:21:38.004001 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0507 11:21:38.004032 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0507 11:21:38.004044 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0507 11:21:38.004049 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0507 11:21:38.004062 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0507 11:21:38.004491 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-317971646/tls.crt::/tmp/serving-cert-317971646/tls.key" I0507 11:21:38.005217 1 secure_serving.go:213] Serving securely on [::]:8443 I0507 11:21:38.005441 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0507 11:21:38.008233 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0507 11:21:38.008262 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0507 11:21:38.008369 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0507 11:21:38.012992 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 11:21:38.013010 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0507 11:21:38.017673 1 secretconfigobserver.go:119] support secret does not exist I0507 11:21:38.022331 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 11:21:38.026589 1 secretconfigobserver.go:119] support secret does not exist I0507 11:21:38.029460 1 recorder.go:161] Pruning old reports every 7h40m47s, max age is 288h0m0s I0507 11:21:38.034392 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0507 11:21:38.034409 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0507 11:21:38.034421 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0507 11:21:38.034434 1 insightsreport.go:296] Starting report retriever I0507 11:21:38.034442 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0507 11:21:38.034450 1 periodic.go:209] Running clusterconfig gatherer I0507 11:21:38.034509 1 tasks_processing.go:45] number of workers: 64 I0507 11:21:38.034535 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 11:21:38.034542 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 11:21:38.034550 1 tasks_processing.go:69] worker 6 listening for tasks. I0507 11:21:38.034557 1 tasks_processing.go:69] worker 2 listening for tasks. I0507 11:21:38.034544 1 tasks_processing.go:69] worker 12 listening for tasks. I0507 11:21:38.034555 1 tasks_processing.go:69] worker 18 listening for tasks. I0507 11:21:38.034564 1 tasks_processing.go:69] worker 15 listening for tasks. I0507 11:21:38.034568 1 tasks_processing.go:69] worker 5 listening for tasks. I0507 11:21:38.034570 1 tasks_processing.go:69] worker 8 listening for tasks. I0507 11:21:38.034564 1 tasks_processing.go:69] worker 7 listening for tasks. I0507 11:21:38.034573 1 tasks_processing.go:69] worker 13 listening for tasks. I0507 11:21:38.034576 1 tasks_processing.go:69] worker 9 listening for tasks. I0507 11:21:38.034578 1 tasks_processing.go:71] worker 13 working on metrics task. I0507 11:21:38.034579 1 tasks_processing.go:71] worker 7 working on image_pruners task. I0507 11:21:38.034580 1 tasks_processing.go:69] worker 17 listening for tasks. I0507 11:21:38.034586 1 tasks_processing.go:69] worker 11 listening for tasks. I0507 11:21:38.034581 1 tasks_processing.go:69] worker 10 listening for tasks. I0507 11:21:38.034572 1 tasks_processing.go:69] worker 16 listening for tasks. I0507 11:21:38.034581 1 tasks_processing.go:69] worker 4 listening for tasks. I0507 11:21:38.034580 1 tasks_processing.go:69] worker 14 listening for tasks. I0507 11:21:38.034591 1 tasks_processing.go:69] worker 45 listening for tasks. I0507 11:21:38.034594 1 tasks_processing.go:69] worker 59 listening for tasks. I0507 11:21:38.034596 1 tasks_processing.go:69] worker 52 listening for tasks. I0507 11:21:38.034602 1 tasks_processing.go:69] worker 46 listening for tasks. I0507 11:21:38.034614 1 tasks_processing.go:71] worker 6 working on networks task. I0507 11:21:38.034621 1 tasks_processing.go:71] worker 9 working on crds task. I0507 11:21:38.034601 1 tasks_processing.go:69] worker 53 listening for tasks. I0507 11:21:38.034634 1 tasks_processing.go:71] worker 53 working on storage_cluster task. I0507 11:21:38.034688 1 tasks_processing.go:71] worker 16 working on machine_sets task. I0507 11:21:38.034722 1 tasks_processing.go:69] worker 3 listening for tasks. I0507 11:21:38.034730 1 tasks_processing.go:69] worker 61 listening for tasks. I0507 11:21:38.034737 1 tasks_processing.go:69] worker 39 listening for tasks. I0507 11:21:38.034739 1 tasks_processing.go:69] worker 26 listening for tasks. I0507 11:21:38.034611 1 tasks_processing.go:69] worker 19 listening for tasks. I0507 11:21:38.034611 1 tasks_processing.go:69] worker 54 listening for tasks. W0507 11:21:38.034610 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:38.034761 1 tasks_processing.go:69] worker 56 listening for tasks. I0507 11:21:38.034763 1 tasks_processing.go:69] worker 23 listening for tasks. I0507 11:21:38.034767 1 tasks_processing.go:69] worker 35 listening for tasks. I0507 11:21:38.034746 1 tasks_processing.go:69] worker 55 listening for tasks. I0507 11:21:38.034776 1 tasks_processing.go:69] worker 57 listening for tasks. I0507 11:21:38.034776 1 tasks_processing.go:69] worker 25 listening for tasks. I0507 11:21:38.034761 1 tasks_processing.go:69] worker 60 listening for tasks. I0507 11:21:38.034784 1 tasks_processing.go:69] worker 58 listening for tasks. I0507 11:21:38.034786 1 tasks_processing.go:71] worker 45 working on qemu_kubevirt_launcher_logs task. I0507 11:21:38.034786 1 tasks_processing.go:71] worker 17 working on validating_webhook_configurations task. I0507 11:21:38.034781 1 tasks_processing.go:69] worker 62 listening for tasks. I0507 11:21:38.034793 1 tasks_processing.go:71] worker 12 working on authentication task. I0507 11:21:38.034792 1 tasks_processing.go:71] worker 10 working on silenced_alerts task. I0507 11:21:38.034800 1 tasks_processing.go:71] worker 15 working on openstack_controlplanes task. I0507 11:21:38.034804 1 tasks_processing.go:69] worker 51 listening for tasks. I0507 11:21:38.034799 1 tasks_processing.go:69] worker 38 listening for tasks. I0507 11:21:38.034800 1 tasks_processing.go:69] worker 48 listening for tasks. I0507 11:21:38.034755 1 tasks_processing.go:69] worker 22 listening for tasks. I0507 11:21:38.034756 1 tasks_processing.go:69] worker 34 listening for tasks. I0507 11:21:38.034812 1 tasks_processing.go:71] worker 18 working on storage_classes task. I0507 11:21:38.034823 1 tasks_processing.go:69] worker 63 listening for tasks. I0507 11:21:38.034823 1 tasks_processing.go:71] worker 4 working on nodenetworkstates task. W0507 11:21:38.034825 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:38.034832 1 tasks_processing.go:69] worker 49 listening for tasks. I0507 11:21:38.034609 1 tasks_processing.go:69] worker 47 listening for tasks. I0507 11:21:38.034769 1 tasks_processing.go:69] worker 24 listening for tasks. I0507 11:21:38.034772 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 182.569µs to process 0 records I0507 11:21:38.034856 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 28.722µs to process 0 records I0507 11:21:38.034902 1 tasks_processing.go:71] worker 24 working on overlapping_namespace_uids task. I0507 11:21:38.034908 1 tasks_processing.go:71] worker 0 working on machine_healthchecks task. I0507 11:21:38.035058 1 tasks_processing.go:69] worker 32 listening for tasks. I0507 11:21:38.035071 1 tasks_processing.go:71] worker 32 working on config_maps task. I0507 11:21:38.035078 1 tasks_processing.go:71] worker 3 working on ingress_certificates task. I0507 11:21:38.034617 1 tasks_processing.go:69] worker 20 listening for tasks. I0507 11:21:38.034781 1 tasks_processing.go:71] worker 11 working on image_registries task. I0507 11:21:38.035154 1 tasks_processing.go:71] worker 5 working on support_secret task. I0507 11:21:38.034618 1 tasks_processing.go:71] worker 52 working on certificate_signing_requests task. I0507 11:21:38.035333 1 tasks_processing.go:69] worker 42 listening for tasks. I0507 11:21:38.035347 1 tasks_processing.go:69] worker 40 listening for tasks. I0507 11:21:38.035351 1 tasks_processing.go:69] worker 43 listening for tasks. I0507 11:21:38.035353 1 tasks_processing.go:69] worker 41 listening for tasks. I0507 11:21:38.035364 1 tasks_processing.go:71] worker 54 working on sap_config task. I0507 11:21:38.035366 1 tasks_processing.go:71] worker 38 working on feature_gates task. I0507 11:21:38.035407 1 tasks_processing.go:71] worker 48 working on openstack_dataplanedeployments task. I0507 11:21:38.035617 1 tasks_processing.go:71] worker 13 working on pod_network_connectivity_checks task. I0507 11:21:38.035654 1 tasks_processing.go:71] worker 61 working on lokistack task. I0507 11:21:38.035737 1 tasks_processing.go:69] worker 44 listening for tasks. I0507 11:21:38.035750 1 tasks_processing.go:71] worker 23 working on clusterroles task. I0507 11:21:38.035788 1 tasks_processing.go:71] worker 35 working on pdbs task. I0507 11:21:38.034788 1 tasks_processing.go:71] worker 2 working on cluster_apiserver task. I0507 11:21:38.034748 1 tasks_processing.go:69] worker 21 listening for tasks. I0507 11:21:38.034791 1 tasks_processing.go:69] worker 37 listening for tasks. I0507 11:21:38.034813 1 tasks_processing.go:71] worker 1 working on olm_operators task. I0507 11:21:38.036132 1 tasks_processing.go:71] worker 14 working on machine_autoscalers task. I0507 11:21:38.036179 1 tasks_processing.go:71] worker 55 working on openshift_machine_api_events task. I0507 11:21:38.036201 1 tasks_processing.go:71] worker 39 working on machine_config_pools task. I0507 11:21:38.036451 1 tasks_processing.go:71] worker 26 working on proxies task. I0507 11:21:38.034997 1 tasks_processing.go:71] worker 57 working on aggregated_monitoring_cr_names task. I0507 11:21:38.036692 1 tasks_processing.go:69] worker 27 listening for tasks. I0507 11:21:38.036750 1 tasks_processing.go:71] worker 19 working on tsdb_status task. W0507 11:21:38.036787 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:38.036720 1 tasks_processing.go:71] worker 56 working on schedulers task. I0507 11:21:38.036939 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 35.395µs to process 0 records I0507 11:21:38.036802 1 tasks_processing.go:69] worker 33 listening for tasks. I0507 11:21:38.036921 1 tasks_processing.go:69] worker 50 listening for tasks. I0507 11:21:38.036977 1 tasks_processing.go:71] worker 63 working on node_logs task. I0507 11:21:38.036978 1 tasks_processing.go:71] worker 49 working on jaegers task. I0507 11:21:38.036991 1 tasks_processing.go:71] worker 34 working on image task. I0507 11:21:38.034782 1 tasks_processing.go:69] worker 36 listening for tasks. I0507 11:21:38.037241 1 tasks_processing.go:71] worker 47 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0507 11:21:38.037330 1 tasks_processing.go:71] worker 22 working on openshift_logging task. I0507 11:21:38.037470 1 tasks_processing.go:71] worker 8 working on machine_configs task. I0507 11:21:38.037512 1 tasks_processing.go:71] worker 59 working on ceph_cluster task. I0507 11:21:38.037589 1 tasks_processing.go:69] worker 30 listening for tasks. I0507 11:21:38.037666 1 tasks_processing.go:69] worker 31 listening for tasks. I0507 11:21:38.037690 1 tasks_processing.go:71] worker 31 working on active_alerts task. I0507 11:21:38.037710 1 tasks_processing.go:69] worker 28 listening for tasks. I0507 11:21:38.037719 1 tasks_processing.go:74] worker 28 stopped. W0507 11:21:38.037715 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:38.037724 1 tasks_processing.go:71] worker 10 working on nodes task. I0507 11:21:38.037734 1 tasks_processing.go:74] worker 31 stopped. I0507 11:21:38.037744 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 33.833µs to process 0 records I0507 11:21:38.037761 1 tasks_processing.go:69] worker 29 listening for tasks. I0507 11:21:38.037766 1 tasks_processing.go:74] worker 29 stopped. I0507 11:21:38.037772 1 tasks_processing.go:71] worker 58 working on monitoring_persistent_volumes task. I0507 11:21:38.037805 1 tasks_processing.go:71] worker 21 working on sap_pods task. I0507 11:21:38.038294 1 tasks_processing.go:71] worker 33 working on infrastructures task. I0507 11:21:38.038536 1 tasks_processing.go:71] worker 25 working on container_runtime_configs task. I0507 11:21:38.038628 1 tasks_processing.go:71] worker 50 working on oauths task. I0507 11:21:38.038633 1 tasks_processing.go:71] worker 20 working on openstack_version task. I0507 11:21:38.038639 1 tasks_processing.go:71] worker 42 working on service_accounts task. I0507 11:21:38.034623 1 tasks_processing.go:71] worker 46 working on mutating_webhook_configurations task. I0507 11:21:38.038649 1 tasks_processing.go:71] worker 40 working on operators task. I0507 11:21:38.039108 1 tasks_processing.go:71] worker 62 working on dvo_metrics task. I0507 11:21:38.039160 1 tasks_processing.go:71] worker 60 working on nodenetworkconfigurationpolicies task. I0507 11:21:38.039201 1 tasks_processing.go:71] worker 27 working on cost_management_metrics_configs task. I0507 11:21:38.039211 1 tasks_processing.go:71] worker 19 working on openstack_dataplanenodesets task. I0507 11:21:38.039217 1 tasks_processing.go:71] worker 30 working on ingress task. I0507 11:21:38.039676 1 tasks_processing.go:71] worker 43 working on sap_datahubs task. I0507 11:21:38.039745 1 tasks_processing.go:71] worker 36 working on machines task. I0507 11:21:38.039820 1 tasks_processing.go:71] worker 44 working on install_plans task. I0507 11:21:38.039958 1 tasks_processing.go:71] worker 37 working on container_images task. I0507 11:21:38.040025 1 tasks_processing.go:71] worker 51 working on operators_pods_and_events task. I0507 11:21:38.039745 1 tasks_processing.go:71] worker 41 working on version task. I0507 11:21:38.040506 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0507 11:21:38.040528 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0507 11:21:38.040533 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0507 11:21:38.040538 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0507 11:21:38.040558 1 controller.go:489] The operator is still being initialized I0507 11:21:38.040566 1 controller.go:512] The operator is healthy I0507 11:21:38.042030 1 tasks_processing.go:74] worker 53 stopped. I0507 11:21:38.042043 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 7.384453ms to process 0 records I0507 11:21:38.042216 1 tasks_processing.go:74] worker 16 stopped. I0507 11:21:38.042233 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 7.507941ms to process 0 records I0507 11:21:38.042434 1 tasks_processing.go:74] worker 7 stopped. I0507 11:21:38.042671 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=94a4da81518e5fbb3e08dc5aefbeedca81271a965e2f313527c66b04a438989d I0507 11:21:38.042685 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 7.839943ms to process 1 records I0507 11:21:38.045221 1 tasks_processing.go:74] worker 54 stopped. I0507 11:21:38.045233 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 9.846802ms to process 0 records I0507 11:21:38.045253 1 tasks_processing.go:74] worker 4 stopped. I0507 11:21:38.045262 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 10.42108ms to process 0 records I0507 11:21:38.045471 1 tasks_processing.go:74] worker 15 stopped. I0507 11:21:38.045485 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 10.656167ms to process 0 records I0507 11:21:38.045511 1 tasks_processing.go:74] worker 0 stopped. E0507 11:21:38.045522 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0507 11:21:38.045529 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 10.593442ms to process 0 records I0507 11:21:38.048120 1 tasks_processing.go:74] worker 59 stopped. I0507 11:21:38.048133 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 10.544321ms to process 0 records I0507 11:21:38.048850 1 tasks_processing.go:74] worker 13 stopped. E0507 11:21:38.048862 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0507 11:21:38.048870 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 13.214307ms to process 0 records I0507 11:21:38.048971 1 tasks_processing.go:74] worker 6 stopped. I0507 11:21:38.049185 1 recorder.go:75] Recording config/network with fingerprint=7203daf56bf6062032cd4b09d257d8743961a456fda7f1a1eb6f7614ae051454 I0507 11:21:38.049205 1 gather.go:177] gatherer "clusterconfig" function "networks" took 14.339496ms to process 1 records I0507 11:21:38.049491 1 tasks_processing.go:74] worker 12 stopped. I0507 11:21:38.049711 1 recorder.go:75] Recording config/authentication with fingerprint=bda5bc8bd0388cae506005eaadbd115fad0f3351ea0f5c7aea6db6dc30f37da8 I0507 11:21:38.049723 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 14.683687ms to process 1 records I0507 11:21:38.051941 1 tasks_processing.go:74] worker 36 stopped. E0507 11:21:38.051956 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0507 11:21:38.051967 1 gather.go:177] gatherer "clusterconfig" function "machines" took 12.153072ms to process 0 records I0507 11:21:38.053119 1 tasks_processing.go:74] worker 22 stopped. I0507 11:21:38.053131 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 15.775953ms to process 0 records I0507 11:21:38.053189 1 tasks_processing.go:74] worker 61 stopped. I0507 11:21:38.053201 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 17.5183ms to process 0 records I0507 11:21:38.053213 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 13.988733ms to process 0 records I0507 11:21:38.053226 1 tasks_processing.go:74] worker 19 stopped. I0507 11:21:38.053347 1 tasks_processing.go:74] worker 26 stopped. I0507 11:21:38.053421 1 recorder.go:75] Recording config/proxy with fingerprint=e98c5366a34c748426d394b8225f7fc722870417113c5153434ed26ed4182f0d I0507 11:21:38.053444 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 16.877173ms to process 1 records I0507 11:21:38.053457 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 17.350582ms to process 0 records I0507 11:21:38.053462 1 tasks_processing.go:74] worker 1 stopped. I0507 11:21:38.053533 1 tasks_processing.go:74] worker 49 stopped. I0507 11:21:38.053547 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 16.539938ms to process 0 records I0507 11:21:38.054101 1 tasks_processing.go:74] worker 48 stopped. I0507 11:21:38.054113 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 18.675887ms to process 0 records I0507 11:21:38.055188 1 tasks_processing.go:74] worker 25 stopped. I0507 11:21:38.055201 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 16.629345ms to process 0 records I0507 11:21:38.056277 1 tasks_processing.go:74] worker 21 stopped. I0507 11:21:38.056288 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 18.454177ms to process 0 records I0507 11:21:38.056818 1 tasks_processing.go:74] worker 34 stopped. I0507 11:21:38.056957 1 recorder.go:75] Recording config/image with fingerprint=a53d9e194c14051f6d422a9ecd6a670ce40eefeffa7032944be9b697c0bc48ed I0507 11:21:38.056972 1 gather.go:177] gatherer "clusterconfig" function "image" took 19.814306ms to process 1 records I0507 11:21:38.058000 1 tasks_processing.go:74] worker 55 stopped. I0507 11:21:38.058011 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 21.807883ms to process 0 records I0507 11:21:38.071478 1 tasks_processing.go:74] worker 2 stopped. I0507 11:21:38.071688 1 recorder.go:75] Recording config/apiserver with fingerprint=09aef08a94302b55c5ab4cd4e62423bf0678d6385613ecf134eaad6233008e65 I0507 11:21:38.071714 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 35.420087ms to process 1 records I0507 11:21:38.071783 1 tasks_processing.go:74] worker 10 stopped. I0507 11:21:38.072210 1 recorder.go:75] Recording config/node/ip-10-0-0-74.ec2.internal with fingerprint=b48e9f3f4dbae14565f35055e18cfa0d84155c3662f1186966908a868c238b9a I0507 11:21:38.072372 1 recorder.go:75] Recording config/node/ip-10-0-2-123.ec2.internal with fingerprint=4332304e8a2d139b3f9c9c2c12acaeb473687f438038e2fc49c61e5e0042c87b I0507 11:21:38.072413 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 33.789846ms to process 2 records I0507 11:21:38.072430 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 34.633768ms to process 0 records I0507 11:21:38.072438 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 35.57937ms to process 0 records I0507 11:21:38.072443 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 33.955622ms to process 0 records I0507 11:21:38.072595 1 recorder.go:75] Recording config/featuregate with fingerprint=46b6e40d4f5a6e6a0c6633d0c96bc5817993dbf0ab99336e9932676bbb1c7a2a I0507 11:21:38.072604 1 tasks_processing.go:74] worker 58 stopped. I0507 11:21:38.072605 1 tasks_processing.go:74] worker 63 stopped. I0507 11:21:38.072616 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 36.540871ms to process 1 records I0507 11:21:38.072612 1 tasks_processing.go:74] worker 38 stopped. I0507 11:21:38.072600 1 tasks_processing.go:74] worker 14 stopped. I0507 11:21:38.072683 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=9db92f0987d649f35b5dac73062539350b65390d1ecff15a4bc6da3dc6a75c9d I0507 11:21:38.072689 1 tasks_processing.go:74] worker 56 stopped. I0507 11:21:38.072694 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 35.084719ms to process 1 records I0507 11:21:38.072775 1 tasks_processing.go:74] worker 11 stopped. I0507 11:21:38.073227 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=2e62a56e65c4a3422080eb7e9d8587fbc0f89f2ff9718358eb17e9d889d75c92 I0507 11:21:38.073250 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.430212ms to process 1 records I0507 11:21:38.073269 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 33.418683ms to process 0 records I0507 11:21:38.073284 1 tasks_processing.go:74] worker 60 stopped. I0507 11:21:38.073350 1 tasks_processing.go:74] worker 30 stopped. I0507 11:21:38.073424 1 recorder.go:75] Recording config/ingress with fingerprint=1c237757922e912d399950c5b9fbf48726367d395e0b8bddb3a16f8f69e2b114 I0507 11:21:38.073436 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 33.379484ms to process 1 records I0507 11:21:38.073523 1 tasks_processing.go:74] worker 33 stopped. I0507 11:21:38.075030 1 recorder.go:75] Recording config/infrastructure with fingerprint=98fe53d63b0fbedd198e47891ec77ab957358ff50b563aeee45608d82fa954bf I0507 11:21:38.075287 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 34.281356ms to process 1 records I0507 11:21:38.075510 1 tasks_processing.go:74] worker 35 stopped. I0507 11:21:38.075551 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=5aaccb4a9436bf6c7ae3e03d0e8e40d501666647536a56980e3ca63b236fb079 I0507 11:21:38.075652 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=aead8310efe542cbdf066bbf00cb3f3896fdc93e782b04325f6d547da08cfeb8 I0507 11:21:38.075794 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=5d3dce629a13ad1d6040c248a048c94a01a19b5d917f2058a29eff9d60569c83 I0507 11:21:38.075862 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 37.143908ms to process 3 records I0507 11:21:38.075803 1 gather_logs.go:145] no pods in namespace were found I0507 11:21:38.075912 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 33.414741ms to process 0 records I0507 11:21:38.075921 1 tasks_processing.go:74] worker 43 stopped. I0507 11:21:38.076425 1 tasks_processing.go:74] worker 20 stopped. I0507 11:21:38.076470 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 34.209503ms to process 0 records I0507 11:21:38.076599 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=616fd4919589bf3535eb99d6779aecb4312f878b6333b76438d6106f91d34c73 I0507 11:21:38.076605 1 tasks_processing.go:74] worker 18 stopped. I0507 11:21:38.076622 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=2f8041b9067cc9943ce0e33d0ceec544ca520a10024ced64af82b1230d198c6d I0507 11:21:38.076634 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 40.702823ms to process 2 records E0507 11:21:38.076643 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0507 11:21:38.076650 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 40.654661ms to process 0 records I0507 11:21:38.076665 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 41.121315ms to process 0 records I0507 11:21:38.076674 1 tasks_processing.go:74] worker 45 stopped. I0507 11:21:38.076679 1 tasks_processing.go:74] worker 5 stopped. I0507 11:21:38.076751 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0507 11:21:38.076785 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0507 11:21:38.076850 1 tasks_processing.go:74] worker 39 stopped. I0507 11:21:38.076866 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 40.635958ms to process 0 records W0507 11:21:38.076940 1 operator.go:288] started I0507 11:21:38.076968 1 tasks_processing.go:74] worker 27 stopped. I0507 11:21:38.076976 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0507 11:21:38.076985 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 37.754019ms to process 0 records I0507 11:21:38.077148 1 tasks_processing.go:74] worker 46 stopped. I0507 11:21:38.077326 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=4638df4ce57d6bb36843f4aed713fbb46c3159d9c1d40ae19328c5cf5664be67 I0507 11:21:38.077374 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ca0b085a541f5aaf0eaff0179577a6b1c7def48da1921456898d8b263574a2f7 I0507 11:21:38.077447 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=870b2b88f34b262bb0900f49369aaf475b6080f6b6ca4b6c541555b45aab2e66 I0507 11:21:38.077506 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 38.106868ms to process 3 records I0507 11:21:38.077616 1 tasks_processing.go:74] worker 17 stopped. I0507 11:21:38.077675 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=7c6499641f92e1a53895182768b739fc340e9422501035be879dc3e2d3058243 I0507 11:21:38.077827 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=20a0522cac264845487e6d02858c5681afd032ab5ee3573a28ac468d6ab0642b I0507 11:21:38.077853 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=7ab924ee118fac152007958534bacf8b669330d618ead47e95a697d9230c3bd2 I0507 11:21:38.077910 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=6ef82591b6c492fe03c149a157f79a47275fe5f336f2b4b402857797b5054097 I0507 11:21:38.077972 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=ce2e92cfe6d07144d0db4456ebed2874cafeb4560ca812ab35c792348cc57c42 I0507 11:21:38.078018 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=c43ea1505193e7e4f4204cc5e3a27b962b3fcac3a65285646eab3876c7c7a7c8 I0507 11:21:38.078070 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=a7e3c5d5836c7f0d4f8f93c3eebcecc5f1aa4f1dce0837867800bf84fa621d49 I0507 11:21:38.078115 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=c3cf1f39cb1eedd299c6e59e9056ca05de8e4c078887ee14ad8313b8f9554f81 I0507 11:21:38.078148 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=b1bc1aa43247537ea3d00ad7710f0af3cf0cf79f0d53d05dc0e445e9e9913f24 I0507 11:21:38.078209 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=f9511fee4deae21e1e6a2329153517981a0f3550d72f339c93db772031f74308 I0507 11:21:38.078236 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=a790f4e2fa3a791c0ccf0be7f90d9295b60d2f995db030b0b0b65f1417b3a16f I0507 11:21:38.078243 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 42.465168ms to process 11 records I0507 11:21:38.078338 1 tasks_processing.go:74] worker 50 stopped. I0507 11:21:38.078455 1 recorder.go:75] Recording config/oauth with fingerprint=089947a265f0e9583c99fd1358def6d7d0d64024429323de8d70592e803d8565 I0507 11:21:38.078482 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 38.828585ms to process 1 records I0507 11:21:38.078496 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0507 11:21:38.078502 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 42.892273ms to process 1 records I0507 11:21:38.078543 1 tasks_processing.go:74] worker 24 stopped. I0507 11:21:38.078612 1 tasks_processing.go:74] worker 9 stopped. I0507 11:21:38.079180 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=d3190b8b9c55b0fa2a903cd298e8b7d8288d7e7877a4d8b2dffe7b2da6dda118 I0507 11:21:38.079395 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=569498bde20a873c552c5695972ae43b16ab16cff877a329bfc9a708ea9b4bee I0507 11:21:38.079404 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.643985ms to process 2 records I0507 11:21:38.079494 1 tasks_processing.go:74] worker 52 stopped. I0507 11:21:38.079512 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 44.253671ms to process 0 records I0507 11:21:38.087545 1 tasks_processing.go:74] worker 37 stopped. I0507 11:21:38.087630 1 recorder.go:75] Recording config/running_containers with fingerprint=dbf61554f047dca2f309bacd22d8abf9ccaa90e725a6b1147ad0b96ccc33d798 I0507 11:21:38.087647 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 47.363398ms to process 1 records W0507 11:21:38.098026 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 11:21:38.104519 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0507 11:21:38.104509 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0507 11:21:38.104529 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0507 11:21:38.108645 1 base_controller.go:82] Caches are synced for ConfigController I0507 11:21:38.108659 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0507 11:21:38.112263 1 prometheus_rules.go:88] Prometheus rules successfully created I0507 11:21:38.125768 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0507 11:21:38.125782 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0507 11:21:38.125785 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0507 11:21:38.125790 1 controller.go:212] Source scaController *sca.Controller is not ready I0507 11:21:38.125793 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0507 11:21:38.125811 1 controller.go:489] The operator is still being initialized I0507 11:21:38.125817 1 controller.go:512] The operator is healthy I0507 11:21:38.125827 1 tasks_processing.go:74] worker 23 stopped. I0507 11:21:38.126199 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=7fd9dccc7e6b467588cbcc21e4fcbc581c51a5a4b4491af638e3e48b1c379327 I0507 11:21:38.126295 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=9bcf05fb33a1929dbf96c9d1029947a2c40ace52f8997927a606b08cc5bbb79d I0507 11:21:38.126338 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 90.064126ms to process 2 records E0507 11:21:38.134509 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%273538cb34-f347-4a30-9f32-141f1bb78138%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:38380->172.30.0.10:53: read: connection refused I0507 11:21:38.134527 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%273538cb34-f347-4a30-9f32-141f1bb78138%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:38380->172.30.0.10:53: read: connection refused I0507 11:21:38.150611 1 tasks_processing.go:74] worker 57 stopped. I0507 11:21:38.150725 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 114.12053ms to process 0 records I0507 11:21:38.153535 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 11:21:38.163591 1 tasks_processing.go:74] worker 41 stopped. I0507 11:21:38.164102 1 recorder.go:75] Recording config/version with fingerprint=a4eefe269a8747c75aeb7d41ed08800c48aeb70444e841d1b96d455a0e3c85b2 I0507 11:21:38.164121 1 recorder.go:75] Recording config/id with fingerprint=867d3d28665e676920f30ad851386aaed5ff68d4e209c0fa654903c8d9cac19a I0507 11:21:38.164131 1 gather.go:177] gatherer "clusterconfig" function "version" took 123.290891ms to process 2 records I0507 11:21:38.175924 1 tasks_processing.go:74] worker 47 stopped. I0507 11:21:38.175947 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 138.593618ms to process 0 records I0507 11:21:38.177144 1 base_controller.go:82] Caches are synced for LoggingSyncer I0507 11:21:38.177157 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0507 11:21:38.181638 1 tasks_processing.go:74] worker 8 stopped. I0507 11:21:38.181680 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0507 11:21:38.181695 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 144.144617ms to process 1 records I0507 11:21:38.203036 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0507 11:21:38.206316 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:48477->172.30.0.10:53: read: connection refused I0507 11:21:38.206334 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:48477->172.30.0.10:53: read: connection refused I0507 11:21:38.219425 1 tasks_processing.go:74] worker 32 stopped. E0507 11:21:38.219443 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0507 11:21:38.219450 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0507 11:21:38.219455 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0507 11:21:38.219471 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0507 11:21:38.219500 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0507 11:21:38.219509 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0507 11:21:38.219513 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0507 11:21:38.219518 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0507 11:21:38.219556 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0507 11:21:38.219564 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0507 11:21:38.219570 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 184.33714ms to process 7 records I0507 11:21:38.246168 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0507 11:21:38.259054 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0507 11:21:38.310783 1 tasks_processing.go:74] worker 3 stopped. E0507 11:21:38.310805 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0507 11:21:38.310814 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q527bvkh7gr4u97845sc0tcktnqd3u4-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q527bvkh7gr4u97845sc0tcktnqd3u4-primary-cert-bundle-secret" not found I0507 11:21:38.310903 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=4992eae1c5c1d64b4d41e762384a54642b8ec8010dd43bad788024505500fe78 I0507 11:21:38.310919 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 275.684201ms to process 1 records I0507 11:21:38.489634 1 gather_cluster_operator_pods_and_events.go:121] Found 12 pods with 13 containers I0507 11:21:38.489649 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1935832 bytes I0507 11:21:38.489967 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-kg9zs pod in namespace openshift-dns (previous: false). I0507 11:21:38.717783 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kg9zs pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-kg9zs\" is waiting to start: ContainerCreating" I0507 11:21:38.717807 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-kg9zs\" is waiting to start: ContainerCreating" I0507 11:21:38.717820 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-kg9zs pod in namespace openshift-dns (previous: false). I0507 11:21:38.893009 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kg9zs pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-kg9zs\" is waiting to start: ContainerCreating" I0507 11:21:38.893027 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-kg9zs\" is waiting to start: ContainerCreating" I0507 11:21:38.893038 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-csngl pod in namespace openshift-dns (previous: false). W0507 11:21:39.097679 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 11:21:39.114067 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:39.114091 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-x6c6v pod in namespace openshift-dns (previous: false). I0507 11:21:39.296729 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:39.296750 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-8568986865-pkrtd pod in namespace openshift-image-registry (previous: false). I0507 11:21:39.493329 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-8568986865-pkrtd pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-8568986865-pkrtd\" is waiting to start: ContainerCreating" I0507 11:21:39.493355 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-8568986865-pkrtd\" is waiting to start: ContainerCreating" I0507 11:21:39.493367 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-cbf9b898-djl5g pod in namespace openshift-image-registry (previous: false). I0507 11:21:39.691832 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:39.691851 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-cbf9b898-tz4jq pod in namespace openshift-image-registry (previous: false). I0507 11:21:39.737128 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0507 11:21:39.887610 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:39.887632 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-5n5zv pod in namespace openshift-image-registry (previous: false). I0507 11:21:40.096216 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:40.096238 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-kvr4n pod in namespace openshift-image-registry (previous: false). W0507 11:21:40.098099 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 11:21:40.116741 1 tasks_processing.go:74] worker 40 stopped. I0507 11:21:40.116800 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=1ea1ff749bc465e639a749424a49ef77de69f77e8d1b8ff2458378800e39e966 I0507 11:21:40.116838 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=22dd17e0a9777180177d9f63023ce074cb728790cef28272e8b5448d2c539bd9 I0507 11:21:40.116871 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0507 11:21:40.116925 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=f42b45382a792203d525151eca13e9b3004ef1fc8910a0b72a9811c56369cb96 I0507 11:21:40.116943 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0507 11:21:40.116968 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=2ff8ab11dfbdf7670ff6f6c0112dc9c20c734387ef81bd91aaa14e1f5df463d9 I0507 11:21:40.117003 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=17bbcea8ca60cec26d19d50cacef89ba4362cf8be69401a93ac301f1871d3308 I0507 11:21:40.117044 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=fe83e58572d54298a40d84984b6a85fa21040cd6e1b0cffd7e16aa7232aec76f I0507 11:21:40.117079 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=14c066091ec94074f4af6775eae2dbd9a5aa1176a0d16665eaa2164fafae445e I0507 11:21:40.117090 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0507 11:21:40.117109 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=1c63ceeb3ae89d57b2553a147ef2d3ed62b747ac071b37ff36b4db1689794094 I0507 11:21:40.117118 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0507 11:21:40.117141 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=3865e4eccef73edd41a610fd8298d7c91421a9b450166209b9b05c40cd2964c2 I0507 11:21:40.117153 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0507 11:21:40.117169 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=57b8ccd94f988a34b7e093b44e51674f18c1faed3728c18ad5f62eeed0d0d235 I0507 11:21:40.117178 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0507 11:21:40.117191 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=aa3cd7b9717a5ce7bd343e3ed1f0c465327772902e58359090071f5b2383b00d I0507 11:21:40.117202 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0507 11:21:40.117222 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=6b8e237d2770599c36d4e36867d8ebf3c19da3210303129c00ba0b0160b4d600 I0507 11:21:40.117395 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=f7733fac6a73f1411ea8e725c4af4cef19d11443b7b95fc21bbfc81f109e1aa4 I0507 11:21:40.117408 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0507 11:21:40.117417 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0507 11:21:40.117439 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0507 11:21:40.117468 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=cb382b77280a6e1ef69b2f68d154ff473520d61674d4ff1ed73331a45b0efd1d I0507 11:21:40.117493 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=8f44c0214897f793b3693cfdd90880e74ac07d9ca2eb265e85812aa654dbd037 I0507 11:21:40.117503 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0507 11:21:40.117524 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=69fb209a7c2b55e59021300cbf68854ba4aa0c13c8805847b472db81923181cf I0507 11:21:40.117537 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0507 11:21:40.117550 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=f3a6b9a616c5b6e8bf4054614a34b888a60318398fa3c69d673d6a5746ca4b02 I0507 11:21:40.117565 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=cf9a2f3a5fc427bfbff73ffee41f51888f653d76752b32a3b897d5709f0375d8 I0507 11:21:40.117580 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=0548008c44482d7894f5b85e252ce48a96a2f9cdc30d7294c12a7a031596f261 I0507 11:21:40.117598 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=c709e79960915bf067aea21823c260825d49b60307919e07add7ea58bcb84237 I0507 11:21:40.117616 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=792c8a8bb59b0102aa16a6821991dab2046e141963ccfe69aa366038821239e7 I0507 11:21:40.117648 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=4942f97bbd901894aadf420ecfc544435753032926963700bf372ab62eecd741 I0507 11:21:40.117664 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0507 11:21:40.117673 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0507 11:21:40.117684 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.077652493s to process 36 records I0507 11:21:40.294057 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:40.294077 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6fcc4c5b4b-5zkqw pod in namespace openshift-ingress (previous: false). I0507 11:21:40.492899 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6fcc4c5b4b-5zkqw pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6fcc4c5b4b-5zkqw\" is waiting to start: ContainerCreating" I0507 11:21:40.492917 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6fcc4c5b4b-5zkqw\" is waiting to start: ContainerCreating" I0507 11:21:40.492930 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6fcc4c5b4b-q7mvg pod in namespace openshift-ingress (previous: false). I0507 11:21:40.687962 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 11:21:40.687983 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-86b8647445-n4z2r pod in namespace openshift-ingress (previous: false). I0507 11:21:40.893141 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-86b8647445-n4z2r pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-86b8647445-n4z2r\" is waiting to start: ContainerCreating" I0507 11:21:40.893159 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-86b8647445-n4z2r\" is waiting to start: ContainerCreating" I0507 11:21:40.893170 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-w8626 pod in namespace openshift-ingress-canary (previous: false). I0507 11:21:41.094360 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-w8626 pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-w8626\" is waiting to start: ContainerCreating" I0507 11:21:41.094378 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-w8626\" is waiting to start: ContainerCreating" I0507 11:21:41.094394 1 tasks_processing.go:74] worker 51 stopped. I0507 11:21:41.094479 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=2eb83705d8ca261775c6b88428cab546d8bd92006905b902be88a51f2e72350f I0507 11:21:41.094517 1 recorder.go:75] Recording events/openshift-dns with fingerprint=9845d053dcbfa5237fe8e00d985fa0910a2ee743c823f90da78e7dce42d5607d I0507 11:21:41.094600 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=77153abf650e5afb433f7f11f287b35e5f8ac4f506df788237e33d38c1fbfc48 I0507 11:21:41.094634 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=a8f8c29192f565eaec4c8254fb71fd75154e336ccb302f6bd4b6eed76413c3cf I0507 11:21:41.094677 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=6014689af9fbc88b139d9f303cd777138a295c4adcb0a6cbdc28fe8d662f6242 I0507 11:21:41.094689 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=340ac9963491c34946404f83f0d178a0a1228f9697ddea4219813236c38e11f9 I0507 11:21:41.094697 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.054174339s to process 6 records W0507 11:21:41.098095 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0507 11:21:42.097618 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0507 11:21:43.094384 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0507 11:21:43.094414 1 tasks_processing.go:74] worker 62 stopped. E0507 11:21:43.094426 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0507 11:21:43.094437 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0507 11:21:43.094452 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0507 11:21:43.094462 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.055266542s to process 1 records I0507 11:21:50.691006 1 tasks_processing.go:74] worker 44 stopped. I0507 11:21:50.691065 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0507 11:21:50.691083 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.651078928s to process 1 records I0507 11:21:51.210431 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 11:21:51.446669 1 tasks_processing.go:74] worker 42 stopped. I0507 11:21:51.446952 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=8134d5e41c8b88cd9e4cb52a44a4aaa7dc9c85df1ecff9db5f654f5b009336ee I0507 11:21:51.446970 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.40770715s to process 1 records E0507 11:21:51.447035 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.412s with: function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0507 11:21:51.448143 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0507 11:21:51.448158 1 periodic.go:209] Running workloads gatherer I0507 11:21:51.448173 1 tasks_processing.go:45] number of workers: 2 I0507 11:21:51.448179 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 11:21:51.448183 1 tasks_processing.go:71] worker 1 working on workload_info task. I0507 11:21:51.448191 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 11:21:51.448211 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0507 11:21:51.471800 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 20s for image data I0507 11:21:51.476350 1 tasks_processing.go:74] worker 0 stopped. I0507 11:21:51.476371 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 28.114523ms to process 0 records I0507 11:21:51.481118 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0507 11:21:51.490110 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (9ms) I0507 11:21:51.501536 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (11ms) I0507 11:21:51.510554 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (9ms) I0507 11:21:51.519513 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (9ms) I0507 11:21:51.528053 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (9ms) I0507 11:21:51.536988 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (9ms) I0507 11:21:51.545856 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (9ms) I0507 11:21:51.556646 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (11ms) I0507 11:21:51.565210 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (9ms) I0507 11:21:51.580502 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (15ms) I0507 11:21:51.681617 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (101ms) I0507 11:21:51.781615 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0507 11:21:51.881535 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0507 11:21:51.984088 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (103ms) I0507 11:21:52.074706 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 11:21:52.081083 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (97ms) I0507 11:21:52.180600 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (99ms) I0507 11:21:52.275945 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 11:21:52.281509 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (101ms) I0507 11:21:52.383734 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (102ms) I0507 11:21:52.481985 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (98ms) I0507 11:21:52.581734 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0507 11:21:52.681775 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0507 11:21:52.781735 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0507 11:21:52.881732 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0507 11:21:52.981538 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0507 11:21:53.081623 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0507 11:21:53.181839 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0507 11:21:53.282067 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0507 11:21:53.381183 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (99ms) I0507 11:21:53.480906 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0507 11:21:53.581671 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (101ms) I0507 11:21:53.690424 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (109ms) I0507 11:21:53.782151 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (92ms) I0507 11:21:53.782184 1 tasks_processing.go:74] worker 1 stopped. E0507 11:21:53.782195 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0507 11:21:53.782434 1 recorder.go:75] Recording config/workload_info with fingerprint=aef038d0e66a0f08a29df0dae573be3ca3926ab1a66e09e74fa6785b5385196a I0507 11:21:53.782449 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.333994611s to process 1 records E0507 11:21:53.782476 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.334s with: function \"workload_info\" failed with an error" I0507 11:21:53.783587 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0507 11:21:53.783600 1 periodic.go:209] Running conditional gatherer I0507 11:21:53.790349 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0507 11:21:53.797778 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.20:33542->172.30.0.10:53: read: connection refused E0507 11:21:53.798057 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0507 11:21:53.798117 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0507 11:21:53.805036 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0507 11:21:53.805052 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805058 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805061 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805064 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805067 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805070 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805073 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805076 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0507 11:21:53.805078 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0507 11:21:53.805094 1 tasks_processing.go:45] number of workers: 3 I0507 11:21:53.805104 1 tasks_processing.go:69] worker 2 listening for tasks. I0507 11:21:53.805108 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0507 11:21:53.805116 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 11:21:53.805129 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0507 11:21:53.805135 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0507 11:21:53.805132 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 11:21:53.805148 1 tasks_processing.go:74] worker 1 stopped. I0507 11:21:53.805194 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0507 11:21:53.805222 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 964ns to process 1 records I0507 11:21:53.805255 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0507 11:21:53.805263 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.437µs to process 1 records I0507 11:21:53.805269 1 tasks_processing.go:74] worker 0 stopped. I0507 11:21:53.805410 1 tasks_processing.go:74] worker 2 stopped. I0507 11:21:53.805424 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 268.767µs to process 0 records I0507 11:21:53.805453 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.20:33542->172.30.0.10:53: read: connection refused I0507 11:21:53.805473 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0507 11:21:53.833307 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=fc46734e7d7012e9999861fe18c4489ff1d5d985bb089342578741a325c4861c I0507 11:21:53.833461 1 diskrecorder.go:70] Writing 99 records to /var/lib/insights-operator/insights-2026-05-07-112153.tar.gz I0507 11:21:53.839406 1 diskrecorder.go:51] Wrote 99 records to disk in 6ms I0507 11:21:53.839445 1 periodic.go:278] Gathering cluster info every 2h0m0s I0507 11:21:53.839467 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0507 11:22:05.245604 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 11:23:02.637119 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="10baf3cc0cde853f09db32a2f606f7d3d1a884c724d2aa0e2f6e1f593fc1fd6c") W0507 11:23:02.637155 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0507 11:23:02.637199 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="3b3529ff88434a11b98c36a516681a22882c288f3de6c72bd9bb5a08e1cd78c7") I0507 11:23:02.637272 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0507 11:23:02.637304 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0507 11:23:02.637324 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0507 11:23:02.637275 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="a4fb7102305bf5f6758660240d24347000e22b224e49bd8e4189186aa208110d") E0507 11:23:02.637373 1 controller.go:299] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0507 11:23:02.637376 1 base_controller.go:181] Shutting down ConfigController ... I0507 11:23:02.637390 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0507 11:23:02.637398 1 base_controller.go:123] Shutting down worker of ConfigController controller ... E0507 11:23:02.637401 1 controller.go:299] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0507 11:23:02.637405 1 base_controller.go:113] All ConfigController workers have been terminated I0507 11:23:02.637410 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0507 11:23:02.637423 1 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0507 11:23:02.637424 1 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/tmp/serving-cert-317971646/tls.crt::/tmp/serving-cert-317971646/tls.key" I0507 11:23:02.637434 1 periodic.go:170] Shutting down I0507 11:23:02.637429 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0507 11:23:02.637446 1 base_controller.go:181] Shutting down LoggingSyncer ... I0507 11:23:02.637455 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... E0507 11:23:02.637426 1 controller.go:299] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0507 11:23:02.637463 1 base_controller.go:113] All LoggingSyncer workers have been terminated