W0521 19:31:38.279317 1 cmd.go:257] Using insecure, self-signed certificates I0521 19:31:38.938583 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:38.939031 1 observer_polling.go:159] Starting file observer I0521 19:31:39.620610 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:31:39.620816 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:31:39.621512 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:31:39.621537 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:31:39.621541 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0521 19:31:39.621537 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0521 19:31:39.621547 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:31:39.621670 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:31:39.621710 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:31:39.621713 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:31:39.624661 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0521 19:31:39.624729 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"c441383a-982a-42f9-93e3-ef10860ffcd2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0521 19:31:39.626670 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:31:39.626688 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:31:39.626707 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:31:39.626709 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:31:39.626727 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:31:39.626731 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:31:39.627073 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-2933370115/tls.crt::/tmp/serving-cert-2933370115/tls.key" I0521 19:31:39.627353 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:31:39.627385 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:31:39.636953 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:31:39.636983 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:31:39.637091 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:31:39.641844 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:31:39.641879 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:31:39.646944 1 secretconfigobserver.go:119] support secret does not exist I0521 19:31:39.651661 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:31:39.655986 1 secretconfigobserver.go:119] support secret does not exist I0521 19:31:39.659475 1 recorder.go:176] Pruning old reports every 5h31m5s, max age is 288h0m0s I0521 19:31:39.664474 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:31:39.664477 1 periodic.go:216] Running clusterconfig gatherer I0521 19:31:39.664493 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:31:39.664515 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:31:39.664520 1 insightsreport.go:296] Starting report retriever I0521 19:31:39.664524 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:31:39.664536 1 tasks_processing.go:45] number of workers: 64 I0521 19:31:39.664565 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:31:39.664576 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:31:39.664585 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:31:39.664582 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:31:39.664590 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:31:39.664600 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:31:39.664605 1 tasks_processing.go:71] worker 8 working on oauths task. I0521 19:31:39.664608 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:31:39.664615 1 tasks_processing.go:71] worker 10 working on config_maps task. I0521 19:31:39.664620 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:31:39.664624 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:31:39.664604 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:31:39.664608 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:31:39.664616 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:31:39.664600 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:31:39.664616 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:31:39.664647 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:31:39.664625 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:31:39.664632 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:31:39.664657 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:31:39.664657 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:31:39.664661 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:31:39.664638 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:31:39.664668 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:31:39.664640 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:31:39.664667 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:31:39.664678 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:31:39.664679 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:31:39.664680 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:31:39.664688 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:31:39.664688 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:31:39.664642 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:31:39.664695 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:31:39.664697 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:31:39.664712 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:31:39.664720 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:31:39.664720 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:31:39.664723 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:31:39.664722 1 tasks_processing.go:71] worker 1 working on olm_operators task. I0521 19:31:39.664723 1 tasks_processing.go:71] worker 12 working on certificate_signing_requests task. I0521 19:31:39.664607 1 tasks_processing.go:71] worker 4 working on image task. I0521 19:31:39.665232 1 tasks_processing.go:71] worker 17 working on node_features task. I0521 19:31:39.665626 1 tasks_processing.go:71] worker 53 working on nodenetworkconfigurationpolicies task. I0521 19:31:39.665754 1 tasks_processing.go:71] worker 5 working on ingress task. I0521 19:31:39.665773 1 tasks_processing.go:71] worker 42 working on machine_healthchecks task. I0521 19:31:39.665923 1 tasks_processing.go:71] worker 43 working on support_secret task. I0521 19:31:39.666011 1 tasks_processing.go:71] worker 24 working on lokistack task. I0521 19:31:39.664649 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:31:39.666367 1 tasks_processing.go:71] worker 32 working on node_logs task. I0521 19:31:39.666473 1 tasks_processing.go:71] worker 6 working on container_images task. I0521 19:31:39.666620 1 tasks_processing.go:71] worker 11 working on storage_classes task. I0521 19:31:39.664625 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:31:39.664652 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:31:39.664639 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:31:39.664634 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:31:39.664664 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:31:39.664663 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:31:39.664671 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:31:39.664671 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:31:39.664672 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:31:39.664678 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:31:39.664682 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:31:39.664689 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:31:39.664683 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:31:39.664684 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:31:39.664689 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:31:39.664698 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:31:39.664695 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:31:39.664697 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:31:39.664703 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:31:39.664698 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:31:39.664701 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:31:39.664707 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:31:39.664712 1 tasks_processing.go:71] worker 2 working on tsdb_status task. I0521 19:31:39.664713 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:31:39.664710 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:31:39.664716 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:31:39.664709 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:31:39.664720 1 tasks_processing.go:71] worker 0 working on monitoring_persistent_volumes task. I0521 19:31:39.664645 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:31:39.666745 1 tasks_processing.go:71] worker 28 working on jaegers task. I0521 19:31:39.666923 1 tasks_processing.go:71] worker 23 working on image_pruners task. I0521 19:31:39.666929 1 tasks_processing.go:71] worker 7 working on sap_config task. I0521 19:31:39.667240 1 tasks_processing.go:71] worker 37 working on service_accounts task. I0521 19:31:39.667425 1 tasks_processing.go:71] worker 13 working on ingress_certificates task. I0521 19:31:39.666782 1 tasks_processing.go:71] worker 41 working on sap_datahubs task. I0521 19:31:39.667307 1 tasks_processing.go:71] worker 29 working on ceph_cluster task. I0521 19:31:39.666761 1 tasks_processing.go:71] worker 20 working on infrastructures task. I0521 19:31:39.666801 1 tasks_processing.go:71] worker 48 working on cost_management_metrics_configs task. I0521 19:31:39.667310 1 tasks_processing.go:71] worker 57 working on networks task. I0521 19:31:39.667314 1 tasks_processing.go:71] worker 3 working on feature_gates task. I0521 19:31:39.667317 1 tasks_processing.go:71] worker 61 working on cluster_apiserver task. I0521 19:31:39.667376 1 tasks_processing.go:71] worker 59 working on image_registries task. I0521 19:31:39.667419 1 tasks_processing.go:71] worker 9 working on overlapping_namespace_uids task. I0521 19:31:39.667418 1 tasks_processing.go:71] worker 62 working on crds task. I0521 19:31:39.667466 1 tasks_processing.go:71] worker 54 working on openshift_logging task. I0521 19:31:39.667509 1 tasks_processing.go:71] worker 44 working on nodes task. I0521 19:31:39.667513 1 tasks_processing.go:71] worker 51 working on authentication task. I0521 19:31:39.667521 1 tasks_processing.go:71] worker 16 working on mutating_webhook_configurations task. I0521 19:31:39.667505 1 tasks_processing.go:71] worker 31 working on openstack_dataplanedeployments task. I0521 19:31:39.666795 1 tasks_processing.go:71] worker 25 working on machine_sets task. I0521 19:31:39.668612 1 tasks_processing.go:71] worker 40 working on schedulers task. I0521 19:31:39.668624 1 tasks_processing.go:71] worker 38 working on pdbs task. I0521 19:31:39.668635 1 tasks_processing.go:71] worker 19 working on proxies task. I0521 19:31:39.668646 1 tasks_processing.go:71] worker 30 working on operators task. I0521 19:31:39.668662 1 tasks_processing.go:71] worker 36 working on sap_pods task. I0521 19:31:39.668676 1 tasks_processing.go:71] worker 58 working on aggregated_monitoring_cr_names task. I0521 19:31:39.668816 1 tasks_processing.go:71] worker 26 working on install_plans task. W0521 19:31:39.668897 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:39.668945 1 tasks_processing.go:71] worker 2 working on openstack_dataplanenodesets task. I0521 19:31:39.668839 1 tasks_processing.go:71] worker 18 working on openstack_version task. I0521 19:31:39.668992 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 1.309944ms to process 0 records I0521 19:31:39.668851 1 tasks_processing.go:71] worker 49 working on machine_config_pools task. I0521 19:31:39.669118 1 tasks_processing.go:71] worker 33 working on qemu_kubevirt_launcher_logs task. I0521 19:31:39.669131 1 tasks_processing.go:71] worker 63 working on clusterroles task. I0521 19:31:39.669204 1 tasks_processing.go:71] worker 52 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:31:39.668855 1 tasks_processing.go:71] worker 47 working on dvo_metrics task. I0521 19:31:39.669358 1 tasks_processing.go:71] worker 55 working on nodenetworkstates task. I0521 19:31:39.669416 1 tasks_processing.go:71] worker 39 working on metrics task. W0521 19:31:39.669449 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:39.669459 1 tasks_processing.go:71] worker 39 working on operators_pods_and_events task. I0521 19:31:39.669488 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 26.333µs to process 0 records I0521 19:31:39.669502 1 tasks_processing.go:71] worker 60 working on validating_webhook_configurations task. I0521 19:31:39.669495 1 tasks_processing.go:71] worker 35 working on machines task. I0521 19:31:39.669555 1 tasks_processing.go:71] worker 14 working on active_alerts task. W0521 19:31:39.669579 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:39.669589 1 tasks_processing.go:74] worker 14 stopped. I0521 19:31:39.669591 1 tasks_processing.go:71] worker 50 working on container_runtime_configs task. I0521 19:31:39.669598 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 20.575µs to process 0 records I0521 19:31:39.669627 1 tasks_processing.go:71] worker 34 working on machine_autoscalers task. I0521 19:31:39.669681 1 tasks_processing.go:71] worker 56 working on storage_cluster task. I0521 19:31:39.669694 1 tasks_processing.go:71] worker 45 working on pod_network_connectivity_checks task. I0521 19:31:39.669726 1 tasks_processing.go:71] worker 22 working on openshift_machine_api_events task. I0521 19:31:39.669743 1 tasks_processing.go:71] worker 15 working on silenced_alerts task. W0521 19:31:39.669963 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:39.669749 1 tasks_processing.go:71] worker 21 working on machine_configs task. I0521 19:31:39.669977 1 tasks_processing.go:74] worker 15 stopped. I0521 19:31:39.669986 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 35.121µs to process 0 records I0521 19:31:39.669753 1 tasks_processing.go:71] worker 46 working on openstack_controlplanes task. I0521 19:31:39.670078 1 tasks_processing.go:74] worker 24 stopped. I0521 19:31:39.670106 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.011672ms to process 0 records I0521 19:31:39.670205 1 tasks_processing.go:74] worker 42 stopped. I0521 19:31:39.669758 1 tasks_processing.go:71] worker 27 working on version task. E0521 19:31:39.670233 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:31:39.670316 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 4.423581ms to process 0 records I0521 19:31:39.670725 1 tasks_processing.go:74] worker 7 stopped. I0521 19:31:39.670739 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 3.78236ms to process 0 records I0521 19:31:39.671004 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:31:39.671021 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:31:39.671026 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:31:39.671030 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:31:39.671044 1 controller.go:531] The operator is still being initialized I0521 19:31:39.671052 1 controller.go:554] The operator is healthy I0521 19:31:39.679059 1 tasks_processing.go:74] worker 8 stopped. I0521 19:31:39.679660 1 recorder.go:75] Recording config/oauth with fingerprint=70862d7d185e3d343350b3dd7bd43296978820944ce74cade004af52c3299f25 I0521 19:31:39.679677 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 14.42078ms to process 1 records E0521 19:31:39.679685 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:31:39.679689 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 13.183239ms to process 0 records I0521 19:31:39.679721 1 tasks_processing.go:74] worker 43 stopped. I0521 19:31:39.679789 1 recorder.go:75] Recording config/image with fingerprint=7eeb78ea0fa5727110ad7eed554f8960274898dcfa316142a16748c6bbe2038a I0521 19:31:39.679797 1 gather.go:177] gatherer "clusterconfig" function "image" took 14.179826ms to process 1 records I0521 19:31:39.679803 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 14.771689ms to process 0 records I0521 19:31:39.679808 1 tasks_processing.go:74] worker 4 stopped. I0521 19:31:39.679809 1 tasks_processing.go:74] worker 1 stopped. I0521 19:31:39.679895 1 tasks_processing.go:74] worker 5 stopped. I0521 19:31:39.680111 1 recorder.go:75] Recording config/ingress with fingerprint=733a6eb9b0864b0ed9e08253aa28b2b25bcfb6351b94df0aeea027299e778399 I0521 19:31:39.680144 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 14.120391ms to process 1 records I0521 19:31:39.680453 1 tasks_processing.go:74] worker 12 stopped. I0521 19:31:39.680473 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 15.704709ms to process 0 records I0521 19:31:39.696583 1 tasks_processing.go:74] worker 28 stopped. I0521 19:31:39.696602 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 29.663581ms to process 0 records E0521 19:31:39.696612 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0521 19:31:39.696616 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 27.621074ms to process 0 records I0521 19:31:39.696632 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 31.3673ms to process 0 records I0521 19:31:39.696645 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 27.952676ms to process 0 records I0521 19:31:39.696654 1 tasks_processing.go:74] worker 36 stopped. I0521 19:31:39.696656 1 tasks_processing.go:74] worker 17 stopped. I0521 19:31:39.696670 1 tasks_processing.go:74] worker 18 stopped. I0521 19:31:39.696709 1 tasks_processing.go:74] worker 2 stopped. I0521 19:31:39.696717 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 27.747763ms to process 0 records I0521 19:31:39.696759 1 tasks_processing.go:74] worker 55 stopped. I0521 19:31:39.696808 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 27.383565ms to process 0 records I0521 19:31:39.696850 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 31.126709ms to process 0 records I0521 19:31:39.696860 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 27.683982ms to process 0 records I0521 19:31:39.696859 1 tasks_processing.go:74] worker 53 stopped. I0521 19:31:39.696865 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 30.371898ms to process 0 records I0521 19:31:39.696871 1 tasks_processing.go:74] worker 32 stopped. I0521 19:31:39.696873 1 tasks_processing.go:74] worker 49 stopped. I0521 19:31:39.696915 1 tasks_processing.go:74] worker 35 stopped. E0521 19:31:39.696925 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:31:39.696941 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.402405ms to process 0 records I0521 19:31:39.697024 1 recorder.go:75] Recording config/proxy with fingerprint=e5222ee82a1188aab13ae2eb758d0b8f4dfe0a09cc9a4b4fff011023e7204e7d I0521 19:31:39.697033 1 tasks_processing.go:74] worker 19 stopped. I0521 19:31:39.697043 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 28.273854ms to process 1 records I0521 19:31:39.697065 1 tasks_processing.go:74] worker 50 stopped. I0521 19:31:39.697119 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 27.461726ms to process 0 records I0521 19:31:39.697132 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 27.405771ms to process 0 records I0521 19:31:39.697135 1 gather_logs.go:145] no pods in namespace were found I0521 19:31:39.697141 1 tasks_processing.go:74] worker 56 stopped. I0521 19:31:39.697145 1 tasks_processing.go:74] worker 33 stopped. I0521 19:31:39.697154 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 28.011672ms to process 0 records I0521 19:31:39.697174 1 tasks_processing.go:74] worker 34 stopped. I0521 19:31:39.697193 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 27.516279ms to process 0 records I0521 19:31:39.697337 1 tasks_processing.go:74] worker 0 stopped. I0521 19:31:39.697352 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 29.683397ms to process 0 records I0521 19:31:39.697485 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7574bc1b757ee1de987f5134cd3e1c1c402e0ea5e227d00706912e1789526f84 I0521 19:31:39.697484 1 tasks_processing.go:74] worker 11 stopped. I0521 19:31:39.697510 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=7b33a101b3f596f3172e227985ee501160660201bb8ee2808305d2d2a86903cb I0521 19:31:39.697520 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.744089ms to process 2 records I0521 19:31:39.697714 1 recorder.go:75] Recording config/network with fingerprint=00197174d841e5eb366a33ef534c863e98fc574688dba0fa1d92a8d41af0d901 I0521 19:31:39.697726 1 gather.go:177] gatherer "clusterconfig" function "networks" took 29.652339ms to process 1 records I0521 19:31:39.697734 1 tasks_processing.go:74] worker 57 stopped. I0521 19:31:39.697760 1 tasks_processing.go:74] worker 45 stopped. E0521 19:31:39.697767 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:31:39.697776 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 28.058402ms to process 0 records I0521 19:31:39.699585 1 tasks_processing.go:74] worker 31 stopped. I0521 19:31:39.699605 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 31.362588ms to process 0 records I0521 19:31:39.700332 1 tasks_processing.go:74] worker 23 stopped. I0521 19:31:39.700934 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=2d0952d774129aa33931f13f3f510949752042e0c0576eae54dfd7cbb9012f52 I0521 19:31:39.700955 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 33.397581ms to process 1 records I0521 19:31:39.701185 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=2ff2343e909480d0fb2c1a41878c584c6a7031d3256b95f7118c8a0c139b7193 I0521 19:31:39.701256 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=46c6cc6264e0a71ecde140353c58c6ed93f3700f411a226041de8a95a5657b37 I0521 19:31:39.701300 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e070b7e158be037c2316f5bef034e60aeb143c73568357cc58253098d481b227 I0521 19:31:39.701313 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 31.715643ms to process 3 records I0521 19:31:39.701331 1 tasks_processing.go:74] worker 38 stopped. I0521 19:31:39.701628 1 tasks_processing.go:74] worker 40 stopped. I0521 19:31:39.701789 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=8c9df8035cd19f7dc8f48e4834d482f4c701495eb986eef9b6026dc88197ec53 I0521 19:31:39.701887 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 32.995066ms to process 1 records I0521 19:31:39.701912 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 34.182588ms to process 0 records I0521 19:31:39.701920 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 34.163206ms to process 0 records I0521 19:31:39.701928 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 33.886958ms to process 0 records I0521 19:31:39.701943 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 33.644375ms to process 0 records I0521 19:31:39.701955 1 tasks_processing.go:74] worker 25 stopped. I0521 19:31:39.701962 1 tasks_processing.go:74] worker 41 stopped. I0521 19:31:39.701969 1 tasks_processing.go:74] worker 29 stopped. I0521 19:31:39.701968 1 tasks_processing.go:74] worker 46 stopped. I0521 19:31:39.701982 1 tasks_processing.go:74] worker 54 stopped. I0521 19:31:39.701981 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 31.899063ms to process 0 records I0521 19:31:39.702206 1 tasks_processing.go:74] worker 48 stopped. I0521 19:31:39.702259 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 34.419837ms to process 0 records I0521 19:31:39.702996 1 tasks_processing.go:74] worker 51 stopped. I0521 19:31:39.703467 1 recorder.go:75] Recording config/authentication with fingerprint=fae96701a326d3aca9722123df01700c89af6e0a16536239552adedf9f0df238 I0521 19:31:39.703585 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 34.884485ms to process 1 records I0521 19:31:39.703690 1 tasks_processing.go:74] worker 20 stopped. I0521 19:31:39.704891 1 recorder.go:75] Recording config/infrastructure with fingerprint=f900eb44f4e8e3934148fcddd6ada3dc26fdf64d96c7575d4d064b95dec5fe56 I0521 19:31:39.704917 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 35.451198ms to process 1 records I0521 19:31:39.705077 1 tasks_processing.go:74] worker 60 stopped. I0521 19:31:39.705140 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=eb2153fd63b71f8d263bdfd51498319902370ec0cd28c6fe3a3042bc805a07e6 I0521 19:31:39.705303 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=d232627763b049dfd3d06ade8bf6eed6367b4866df6192aee6785206cf69557d I0521 19:31:39.705379 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=ad0242886fa467ef7b2c5c62542a0dfa42c18878f7d613ce4648f08a07847ce7 I0521 19:31:39.705464 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=b169fd4cacb9ad4c29bc2b279033fa9516a7ae8f217f5cabd04c725b9082188d I0521 19:31:39.705522 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=c448e38f8c317e05f20d47c78c9ca8ff87388041659af92ca7168d11bd828189 I0521 19:31:39.705579 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=672283bf5dbd3d7edbaef2615f6f2f5b0214d35b31ebc232f21c293903b515c3 I0521 19:31:39.705629 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=2e9286b3a660b7be32eae47b885302afbfeeb0c2462e7ce787730f33bb42bbe8 I0521 19:31:39.705685 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=a9152a61c7f7a0cfc286517a73c1811e05546bbe6096be4e9ac951a38b7589f4 I0521 19:31:39.705723 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=c2cc0e139b6e09f3404da2b2460b259f898139e903fdc1cd0744f84cfbd3745e I0521 19:31:39.705764 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=1d724c0dd361746727991a899d5e16295d128eea78638f99f5f479fd5b615c00 I0521 19:31:39.705807 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=5bc9cdf06808beca5db08f1c626338d2358d81802b4766a2144e3290ce9a3220 I0521 19:31:39.705826 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 34.942318ms to process 11 records I0521 19:31:39.705841 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 35.215564ms to process 0 records I0521 19:31:39.705858 1 tasks_processing.go:74] worker 22 stopped. I0521 19:31:39.705942 1 tasks_processing.go:74] worker 3 stopped. I0521 19:31:39.705956 1 recorder.go:75] Recording config/featuregate with fingerprint=3a8dcf44ee89926cb46319ca9acca6498131bda49ff88273db096e729523ee30 I0521 19:31:39.705964 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 37.476889ms to process 1 records I0521 19:31:39.706076 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=e39fa50512f4cf966ea68d14c9897e855f1f81d8b6634a62de1b6c81ec072477 I0521 19:31:39.706088 1 tasks_processing.go:74] worker 16 stopped. I0521 19:31:39.706129 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=335033160d268e9a3d087178c8952c39e9bbbee28de6d0f836e66c58c7bdb874 I0521 19:31:39.706205 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=09b235bcb6d48817c56df449cd2f73305f8297494dde025c1f1327073a275f53 I0521 19:31:39.706228 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 37.38075ms to process 3 records I0521 19:31:39.706380 1 tasks_processing.go:74] worker 59 stopped. I0521 19:31:39.706787 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=bdbab66437e5e1aa4de34db5bdac6d38b1158f85388c9e8e53b03ad35ab43888 I0521 19:31:39.706839 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.798062ms to process 1 records I0521 19:31:39.706946 1 tasks_processing.go:74] worker 61 stopped. I0521 19:31:39.707037 1 recorder.go:75] Recording config/apiserver with fingerprint=0fd45015951466974cb837afcc62b28d24c4e07981506faf09ad6e4e994f3fe1 I0521 19:31:39.707054 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 38.111976ms to process 1 records I0521 19:31:39.708230 1 tasks_processing.go:74] worker 21 stopped. E0521 19:31:39.708247 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0521 19:31:39.708289 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:31:39.708301 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 38.238313ms to process 1 records W0521 19:31:39.708437 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:31:39.713159 1 tasks_processing.go:74] worker 63 stopped. I0521 19:31:39.713348 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=6202290c83533a289b4b82b2c8b46d5f245249639d6159737e6fd2db20ccc77a I0521 19:31:39.713425 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:31:39.713497 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0521 19:31:39.713542 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=d61379017261bbfdbbb49928f5f47fb58725e70349706812c4bbac25dbb710ac I0521 19:31:39.713746 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 44.017718ms to process 2 records W0521 19:31:39.713611 1 operator.go:328] started I0521 19:31:39.713634 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0521 19:31:39.715822 1 tasks_processing.go:74] worker 52 stopped. I0521 19:31:39.715854 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 46.602342ms to process 0 records I0521 19:31:39.716798 1 tasks_processing.go:74] worker 62 stopped. I0521 19:31:39.718341 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=a17526f12e298fb14f662e215890489a13a37cc63f5dee619f85e59b0f1a14f5 I0521 19:31:39.718878 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=b4c5d1ac2e260ca0815afbc1c6e7c62e638f75e0715a272c37cae8f046784868 I0521 19:31:39.718938 1 gather.go:177] gatherer "clusterconfig" function "crds" took 48.837546ms to process 2 records I0521 19:31:39.720466 1 tasks_processing.go:74] worker 58 stopped. I0521 19:31:39.720483 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 51.759624ms to process 0 records I0521 19:31:39.725485 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:31:39.725503 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:31:39.725509 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:31:39.725513 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:31:39.725518 1 controller.go:254] Source scaController *sca.Controller is not ready I0521 19:31:39.725543 1 controller.go:531] The operator is still being initialized I0521 19:31:39.725555 1 controller.go:554] The operator is healthy I0521 19:31:39.727208 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:31:39.727223 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:31:39.727242 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:31:39.728445 1 prometheus_rules.go:88] Prometheus rules successfully created I0521 19:31:39.731718 1 tasks_processing.go:74] worker 10 stopped. E0521 19:31:39.731776 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:31:39.731788 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:31:39.731792 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:31:39.731801 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0521 19:31:39.731838 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:31:39.731846 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:31:39.731850 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0521 19:31:39.731854 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:31:39.731920 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:31:39.731932 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:31:39.731939 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 67.057404ms to process 7 records I0521 19:31:39.737448 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:31:39.737460 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... E0521 19:31:39.741287 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%279af8e854-5599-413c-b409-390ff141e8ba%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:57129->172.30.0.10:53: read: connection refused I0521 19:31:39.741303 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%279af8e854-5599-413c-b409-390ff141e8ba%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:57129->172.30.0.10:53: read: connection refused I0521 19:31:39.741946 1 tasks_processing.go:74] worker 13 stopped. E0521 19:31:39.741966 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:31:39.741974 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegno3se8u2r4kek6nimg6l9k2oshgs-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegno3se8u2r4kek6nimg6l9k2oshgs-primary-cert-bundle-secret" not found I0521 19:31:39.742048 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=9c9da3331eec4cd7664a76b03170e32e729e8b79f88a5b1df8dd448ff2184349 I0521 19:31:39.742066 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 74.345887ms to process 1 records I0521 19:31:39.743349 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:31:39.768364 1 tasks_processing.go:74] worker 27 stopped. I0521 19:31:39.768774 1 recorder.go:75] Recording config/version with fingerprint=bce562ba511e4e23afb056b0855a48d65d4106c2db65eb39efbc030657e063d5 I0521 19:31:39.768797 1 recorder.go:75] Recording config/id with fingerprint=f3d9e67754cbd459bb361066085c1b446c2ea12a36106e16a8a8e49d8c3d57d6 I0521 19:31:39.768809 1 gather.go:177] gatherer "clusterconfig" function "version" took 98.095342ms to process 2 records I0521 19:31:39.778761 1 tasks_processing.go:74] worker 6 stopped. I0521 19:31:39.778944 1 recorder.go:75] Recording config/running_containers with fingerprint=c246e2138b12ccc3588a7f632f50815e822a0eaa2dac55d1d53c2e844db971cc I0521 19:31:39.779003 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 112.224254ms to process 1 records I0521 19:31:39.779535 1 tasks_processing.go:74] worker 44 stopped. I0521 19:31:39.779893 1 recorder.go:75] Recording config/node/ip-10-0-0-70.ec2.internal with fingerprint=9372d04e96c5b40a726a009bfe26ff812f8e03ead22a7a68e06a193fbe05e96b I0521 19:31:39.779921 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 111.440995ms to process 1 records I0521 19:31:39.788319 1 tasks_processing.go:74] worker 9 stopped. I0521 19:31:39.788352 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:31:39.788364 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 120.389584ms to process 1 records I0521 19:31:39.796585 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0521 19:31:39.801924 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:48782->172.30.0.10:53: read: connection refused I0521 19:31:39.801942 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:48782->172.30.0.10:53: read: connection refused I0521 19:31:39.814079 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:31:39.814099 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0521 19:31:40.130805 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0521 19:31:40.130822 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0521 19:31:40.131545 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-qlrqt pod in namespace openshift-dns (previous: false). I0521 19:31:40.351141 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-qlrqt pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-qlrqt\" is waiting to start: ContainerCreating" I0521 19:31:40.351167 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-qlrqt\" is waiting to start: ContainerCreating" I0521 19:31:40.351179 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-qlrqt pod in namespace openshift-dns (previous: false). I0521 19:31:40.533036 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-qlrqt pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-qlrqt\" is waiting to start: ContainerCreating" I0521 19:31:40.533054 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-qlrqt\" is waiting to start: ContainerCreating" I0521 19:31:40.533065 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-d6s8n pod in namespace openshift-dns (previous: false). W0521 19:31:40.701790 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:31:40.733786 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:31:40.733811 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-676cd4c8dd-xv2kl pod in namespace openshift-image-registry (previous: false). I0521 19:31:40.933791 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:31:40.933810 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-84478cfc99-bchk8 pod in namespace openshift-image-registry (previous: false). I0521 19:31:41.135116 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0521 19:31:41.138540 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-84478cfc99-bchk8 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-84478cfc99-bchk8\" is waiting to start: ContainerCreating" I0521 19:31:41.138562 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-84478cfc99-bchk8\" is waiting to start: ContainerCreating" I0521 19:31:41.138576 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-84478cfc99-g4sg9 pod in namespace openshift-image-registry (previous: false). I0521 19:31:41.327148 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:31:41.327166 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-tq9dr pod in namespace openshift-image-registry (previous: false). I0521 19:31:41.532668 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:31:41.532695 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-74bf5d879c-cv8xn pod in namespace openshift-ingress (previous: false). I0521 19:31:41.536830 1 tasks_processing.go:74] worker 30 stopped. I0521 19:31:41.536885 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=3072ccba044197be3ecb7f46694f63a5456bb279a3439f848b6e0ed464d84971 I0521 19:31:41.536940 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=8e0145f922a86cbbc7ccb24cdcf05884331c4fe08dcb53a82d7561646ca6cc2f I0521 19:31:41.536978 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:31:41.537006 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=79f52c6be332d8ba7ce001b970fa257c51d06348973849674752e6c112301f99 I0521 19:31:41.537023 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:31:41.537044 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=6483ccf803758c17c4bb3b6d2c74728741d69439c5bca3a451e46a6b54d4745e I0521 19:31:41.537082 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=daefe4ae8eae10480e9d9b10d7ec283c238e65acef437fcd2088ca969ef5c05c I0521 19:31:41.537107 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=0f5692e45908676c34cd55e31a44f654e568384e4c166ee5d25bfe353ff841c6 I0521 19:31:41.537138 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=e5fb6140375e52af3f00446d79d5dc1e29fd78f9c6301a64a5465318c33c5753 I0521 19:31:41.537165 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=ebdbf6f72ca1aa3723ddde9ef7b82d3c1afac6ab68edf277ff4b9f4936f2685a I0521 19:31:41.537177 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:31:41.537194 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=1d60bb28f876c0aa5d7a06efec66987b5529d256cf048784fdbab79e8dfdf7a6 I0521 19:31:41.537206 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:31:41.537221 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=91e4808d64c384252d7332126a8a78116df68545b695986b46f5d6a5c8a865c7 I0521 19:31:41.537230 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:31:41.537249 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=b252671251b4fa1ebb43e7868c30c29bbe948f133f7081c4fc09a4244bfb42c9 I0521 19:31:41.537259 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:31:41.537275 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=fcebaffffd533416446d2ffc411941dfc62afebbaf8e6107158710fe706ce49b I0521 19:31:41.537428 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=6fa5eec924dca41cbb71ff3cba2f6b1b3be0e4f9fba4286773ffcc250f42867f I0521 19:31:41.537440 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:31:41.537448 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:31:41.537481 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:31:41.537507 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=0c72755feaee11f7bde4478d8a7acbf2eb49bf44b4a1f791ea4f418316a20976 I0521 19:31:41.537531 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=c22d5436a471e49632433c64409ec771ab4f12aa13a4f3717a703cdd274e84a2 I0521 19:31:41.537542 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:31:41.537560 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=1604e26396d5da38fc6d03d3a07d8666a2dcd293a9e244f0d18488c385ffc4e4 I0521 19:31:41.537575 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:31:41.537587 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=700d868a191ef56d797e37dbd579d3ecd1183712806b5b515900c34af322633a I0521 19:31:41.537600 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=4a94e329ba6f05a647356f11ad727a111ae869b5fe2075a13043ac11e4ba3b79 I0521 19:31:41.537613 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=6da226e83665ab3a6e270ed0cf8777412dc08a8773cf6e4ddb507f44dab88fd9 I0521 19:31:41.537626 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=6e3cb4dcc3a7aa654456fc6a48cf9de9321a4d2306efe45d8c6b7e983d5d38a9 I0521 19:31:41.537656 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=8a0865aa5955c41c14d3b88bc0a0d0e432204894d692d68927061edf547b5d10 I0521 19:31:41.537688 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=65b9a67e7433da4af0d3ef7cd14dcb457994c48fff599550ab07e6c4d387fb9f I0521 19:31:41.537704 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:31:41.537711 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:31:41.537718 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.868166869s to process 35 records W0521 19:31:41.701811 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:31:41.733199 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-74bf5d879c-cv8xn pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-74bf5d879c-cv8xn\" is waiting to start: ContainerCreating" I0521 19:31:41.733218 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-74bf5d879c-cv8xn\" is waiting to start: ContainerCreating" I0521 19:31:41.733231 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-767949f4c-nllwm pod in namespace openshift-ingress (previous: false). I0521 19:31:41.950452 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-767949f4c-nllwm pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-767949f4c-nllwm\" is waiting to start: ContainerCreating" I0521 19:31:41.950471 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-767949f4c-nllwm\" is waiting to start: ContainerCreating" I0521 19:31:41.950484 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-767949f4c-rlk4w pod in namespace openshift-ingress (previous: false). I0521 19:31:42.127800 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:31:42.127819 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-fnhst pod in namespace openshift-ingress-canary (previous: false). I0521 19:31:42.336589 1 tasks_processing.go:74] worker 39 stopped. I0521 19:31:42.336672 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=f2fb21f185760a78160aa767202e3d2af0bed57da6836beb8efcc9715c48cfaf I0521 19:31:42.336698 1 recorder.go:75] Recording events/openshift-dns with fingerprint=5fb54fc49eb0e97e63115abf10b19d54c6e65880911098dc3486f970060c22ae I0521 19:31:42.336768 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=eeb55e5f4bb7ad9b15d59be0f2d96321d4c17a8c52ca821daf79dfab040830bb I0521 19:31:42.336797 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=99ed610333c052d5f8b8006354500acb66d346c805b029b85763b8c10457e865 I0521 19:31:42.336838 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=8e5ca13dcbcf7f06efea1daec6aa9dc7a0315103f11dc2dcf8766f5358a2e498 I0521 19:31:42.336855 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=de499f962271447ab83b2b4a2f9b4abcbd0cbcaf6d9fca90d429b1be86108a5d I0521 19:31:42.336862 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-fnhst/serve-healthcheck-canary_current.log with fingerprint=6fa278b8bd32907ef2f7b086de7914c7a8b1afec33e8ccec253dea5250901521 I0521 19:31:42.336869 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.667115348s to process 7 records W0521 19:31:42.701965 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:31:43.701594 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:31:44.698082 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:31:44.698129 1 tasks_processing.go:74] worker 47 stopped. E0521 19:31:44.698142 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:31:44.698153 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0521 19:31:44.698168 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0521 19:31:44.698177 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.028802336s to process 1 records I0521 19:31:47.015304 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:31:52.315639 1 tasks_processing.go:74] worker 26 stopped. I0521 19:31:52.315681 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0521 19:31:52.315695 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.646755764s to process 1 records I0521 19:31:53.076527 1 tasks_processing.go:74] worker 37 stopped. I0521 19:31:53.076800 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=2a6a98cd978f6a8f6d561f8f5025ec9397f180ae98d9caca3683e8186873f98f I0521 19:31:53.076818 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.408982342s to process 1 records E0521 19:31:53.076882 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.412s with: function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0521 19:31:53.077991 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0521 19:31:53.078007 1 periodic.go:216] Running workloads gatherer I0521 19:31:53.078025 1 tasks_processing.go:45] number of workers: 2 I0521 19:31:53.078032 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:31:53.078037 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:31:53.078045 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:31:53.078070 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:31:53.100313 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 20s for image data I0521 19:31:53.105803 1 tasks_processing.go:74] worker 0 stopped. I0521 19:31:53.105829 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 27.703532ms to process 0 records I0521 19:31:53.113697 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (14ms) I0521 19:31:53.122484 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (9ms) I0521 19:31:53.131013 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (9ms) I0521 19:31:53.139794 1 gather_workloads_info.go:387] No image sha256:c886c2a85f0c920fcd6927a7c76644a422cf77f07b185120a8021e0e98d6acb9 (9ms) I0521 19:31:53.148018 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (8ms) I0521 19:31:53.155995 1 gather_workloads_info.go:387] No image sha256:3a1c7ec9bb0db343348170a70697ea3209ed81e22f1259ff694190e5d04ef250 (8ms) I0521 19:31:53.164983 1 gather_workloads_info.go:387] No image sha256:f4115eca8ea784ac6037602a303fac7fb808d98f4b5120a27bee140df60c3664 (9ms) I0521 19:31:53.173473 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (8ms) I0521 19:31:53.181905 1 gather_workloads_info.go:387] No image sha256:90a8ffd9643ebb16a6a8c04bb38cf9ed58903e9d3bf836c68f399193db5edaf6 (8ms) I0521 19:31:53.190229 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (8ms) I0521 19:31:53.209011 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (19ms) I0521 19:31:53.317812 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (109ms) I0521 19:31:53.411132 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (93ms) I0521 19:31:53.517116 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (106ms) I0521 19:31:53.609789 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (93ms) I0521 19:31:53.714282 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (104ms) I0521 19:31:53.810182 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (96ms) I0521 19:31:53.912319 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (102ms) I0521 19:31:54.010663 1 gather_workloads_info.go:387] No image sha256:4733236617781e3469ffffb15e4daaa1f14ea8e1c52b426a3787a4f1f2945424 (98ms) I0521 19:31:54.127097 1 gather_workloads_info.go:387] No image sha256:ce138e8cf5b96557b1864ef6f27c2608bdca59be2611804366cef7169c36291e (116ms) I0521 19:31:54.211234 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (84ms) I0521 19:31:54.316282 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (105ms) I0521 19:31:54.409904 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (94ms) I0521 19:31:54.516750 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (107ms) I0521 19:31:54.616118 1 gather_workloads_info.go:387] No image sha256:4b748615a43be416d52c33df03e2cdec89ededded1d6a2ade167c5a955d56e5f (99ms) I0521 19:31:54.718229 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (102ms) I0521 19:31:54.721454 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:31:54.810117 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (92ms) I0521 19:31:54.912904 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (103ms) I0521 19:31:54.918710 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:31:55.009341 1 gather_workloads_info.go:387] No image sha256:dcb3f93d34aa0fdab33eb37949adfaa8fdd11ff3479363bb7735c5ba3eb345ce (96ms) I0521 19:31:55.113116 1 gather_workloads_info.go:387] No image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (104ms) I0521 19:31:55.210925 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (98ms) I0521 19:31:55.319359 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (108ms) I0521 19:31:55.411649 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (92ms) I0521 19:31:55.515638 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (104ms) I0521 19:31:55.613039 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (97ms) I0521 19:31:55.716181 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (103ms) I0521 19:31:55.809738 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (94ms) I0521 19:31:55.908991 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (99ms) I0521 19:31:56.009696 1 gather_workloads_info.go:387] No image sha256:d9ed66918db4dfba8bd354c9ede4d676449fa0eccef649b3d8945cc1da1c60e3 (101ms) I0521 19:31:56.009731 1 tasks_processing.go:74] worker 1 stopped. E0521 19:31:56.009742 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0521 19:31:56.010012 1 recorder.go:75] Recording config/workload_info with fingerprint=0eda7a89febd7c73ff482e04a71e80cbb3d0c99a5fadab4997eb7e7236c84a6e I0521 19:31:56.010029 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.931686923s to process 1 records E0521 19:31:56.010056 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.932s with: function \"workload_info\" failed with an error" I0521 19:31:56.011162 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0521 19:31:56.011175 1 periodic.go:216] Running conditional gatherer I0521 19:31:56.017090 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0521 19:31:56.024027 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.22:57504->172.30.0.10:53: read: connection refused E0521 19:31:56.024274 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:31:56.024333 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:31:56.030298 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0521 19:31:56.030314 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030319 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030322 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030325 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030328 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030331 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030334 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030337 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:31:56.030340 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0521 19:31:56.030354 1 tasks_processing.go:45] number of workers: 3 I0521 19:31:56.030373 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:31:56.030379 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0521 19:31:56.030382 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:31:56.030409 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:31:56.030415 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0521 19:31:56.030418 1 tasks_processing.go:74] worker 1 stopped. I0521 19:31:56.030424 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0521 19:31:56.030477 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0521 19:31:56.030490 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 931ns to process 1 records I0521 19:31:56.030521 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:31:56.030528 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.323µs to process 1 records I0521 19:31:56.030534 1 tasks_processing.go:74] worker 0 stopped. I0521 19:31:56.030674 1 tasks_processing.go:74] worker 2 stopped. I0521 19:31:56.030687 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 241.453µs to process 0 records I0521 19:31:56.030707 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.22:57504->172.30.0.10:53: read: connection refused I0521 19:31:56.030725 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:31:56.054451 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=c3c8f77606de4d2ea8643368b19d72c5073f4bbde46cd2fcbda476cb953f3d0a I0521 19:31:56.054585 1 diskrecorder.go:70] Writing 98 records to /var/lib/insights-operator/insights-2026-05-21-193156.tar.gz I0521 19:31:56.060211 1 diskrecorder.go:51] Wrote 98 records to disk in 5ms I0521 19:31:56.060243 1 periodic.go:285] Gathering cluster info every 2h0m0s I0521 19:31:56.060258 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:32:06.136576 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:33:08.940204 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="9df067eb2da52ff54fdbcdb4979e9493e3926969060cf5094038e34a91c42cec") W0521 19:33:08.940236 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0521 19:33:08.940298 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="5efa9ab7442ca8b87423d7fea9636aed73bcc319576139da8dacc051452941d3") I0521 19:33:08.940310 1 base_controller.go:181] Shutting down LoggingSyncer ... I0521 19:33:08.940311 1 periodic.go:177] Shutting down I0521 19:33:08.940327 1 base_controller.go:181] Shutting down ConfigController ... I0521 19:33:08.940355 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0521 19:33:08.940376 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="60d6737b5c8a6d1b5cb2e34557cc48dd3595ac56841a11fc59a89cef8e198815") I0521 19:33:08.940380 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0521 19:33:08.940421 1 base_controller.go:123] Shutting down worker of ConfigController controller ...