I0703 22:01:20.695586 1 cmd.go:253] Using service-serving-cert provided certificates I0703 22:01:20.697096 1 observer_polling.go:159] Starting file observer I0703 22:01:21.168245 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0703 22:01:21.168504 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0703 22:01:21.169040 1 secure_serving.go:57] Forcing use of http/1.1 only W0703 22:01:21.169061 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0703 22:01:21.169066 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0703 22:01:21.169070 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0703 22:01:21.169072 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0703 22:01:21.169075 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0703 22:01:21.169077 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0703 22:01:21.169453 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0703 22:01:21.174266 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0703 22:01:21.174276 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"8bb6dad6-c8d3-4c06-9a4c-50db16b19a8a", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0703 22:01:21.175324 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0703 22:01:21.175337 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0703 22:01:21.175362 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0703 22:01:21.175372 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0703 22:01:21.175428 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0703 22:01:21.175442 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0703 22:01:21.175657 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0703 22:01:21.175972 1 secure_serving.go:213] Serving securely on [::]:8443 I0703 22:01:21.176000 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0703 22:01:21.179160 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0703 22:01:21.179186 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0703 22:01:21.179294 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0703 22:01:21.183691 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:01:21.183716 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0703 22:01:21.190219 1 secretconfigobserver.go:119] support secret does not exist I0703 22:01:21.194473 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:01:21.198383 1 secretconfigobserver.go:119] support secret does not exist I0703 22:01:21.200992 1 recorder.go:161] Pruning old reports every 4h7m51s, max age is 288h0m0s I0703 22:01:21.205934 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0703 22:01:21.205955 1 periodic.go:209] Running clusterconfig gatherer I0703 22:01:21.205957 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0703 22:01:21.205956 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0703 22:01:21.205974 1 insightsreport.go:296] Starting report retriever I0703 22:01:21.205981 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0703 22:01:21.206158 1 tasks_processing.go:45] number of workers: 32 I0703 22:01:21.206627 1 tasks_processing.go:69] worker 4 listening for tasks. I0703 22:01:21.206728 1 tasks_processing.go:71] worker 4 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0703 22:01:21.206835 1 tasks_processing.go:69] worker 26 listening for tasks. I0703 22:01:21.206859 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 22:01:21.206922 1 tasks_processing.go:69] worker 7 listening for tasks. I0703 22:01:21.206843 1 tasks_processing.go:69] worker 31 listening for tasks. I0703 22:01:21.206736 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 22:01:21.206856 1 tasks_processing.go:69] worker 27 listening for tasks. I0703 22:01:21.206864 1 tasks_processing.go:69] worker 28 listening for tasks. I0703 22:01:21.206897 1 tasks_processing.go:69] worker 5 listening for tasks. I0703 22:01:21.206896 1 tasks_processing.go:69] worker 29 listening for tasks. I0703 22:01:21.206911 1 tasks_processing.go:69] worker 6 listening for tasks. I0703 22:01:21.206940 1 tasks_processing.go:69] worker 16 listening for tasks. I0703 22:01:21.206947 1 tasks_processing.go:69] worker 8 listening for tasks. I0703 22:01:21.207358 1 tasks_processing.go:69] worker 30 listening for tasks. I0703 22:01:21.207382 1 tasks_processing.go:69] worker 21 listening for tasks. I0703 22:01:21.207391 1 tasks_processing.go:69] worker 18 listening for tasks. I0703 22:01:21.207401 1 tasks_processing.go:69] worker 19 listening for tasks. I0703 22:01:21.207403 1 tasks_processing.go:69] worker 17 listening for tasks. I0703 22:01:21.207411 1 tasks_processing.go:69] worker 20 listening for tasks. I0703 22:01:21.207419 1 tasks_processing.go:69] worker 22 listening for tasks. I0703 22:01:21.207422 1 tasks_processing.go:69] worker 2 listening for tasks. I0703 22:01:21.207430 1 tasks_processing.go:69] worker 23 listening for tasks. I0703 22:01:21.207431 1 tasks_processing.go:69] worker 3 listening for tasks. I0703 22:01:21.207441 1 tasks_processing.go:69] worker 24 listening for tasks. I0703 22:01:21.207443 1 tasks_processing.go:69] worker 12 listening for tasks. I0703 22:01:21.207450 1 tasks_processing.go:69] worker 10 listening for tasks. I0703 22:01:21.207465 1 tasks_processing.go:69] worker 11 listening for tasks. I0703 22:01:21.207502 1 tasks_processing.go:69] worker 14 listening for tasks. I0703 22:01:21.207517 1 tasks_processing.go:69] worker 13 listening for tasks. I0703 22:01:21.207643 1 tasks_processing.go:69] worker 25 listening for tasks. I0703 22:01:21.207652 1 tasks_processing.go:69] worker 15 listening for tasks. I0703 22:01:21.207769 1 tasks_processing.go:71] worker 16 working on ingress_certificates task. I0703 22:01:21.207823 1 tasks_processing.go:71] worker 17 working on networks task. I0703 22:01:21.207857 1 tasks_processing.go:71] worker 0 working on monitoring_persistent_volumes task. I0703 22:01:21.208206 1 tasks_processing.go:71] worker 20 working on image_pruners task. I0703 22:01:21.208276 1 tasks_processing.go:71] worker 5 working on sap_pods task. I0703 22:01:21.208535 1 tasks_processing.go:71] worker 22 working on infrastructures task. I0703 22:01:21.208606 1 tasks_processing.go:71] worker 29 working on machines task. I0703 22:01:21.208649 1 tasks_processing.go:71] worker 14 working on qemu_kubevirt_launcher_logs task. I0703 22:01:21.208643 1 tasks_processing.go:71] worker 28 working on operators task. I0703 22:01:21.207771 1 tasks_processing.go:71] worker 15 working on olm_operators task. I0703 22:01:21.207797 1 tasks_processing.go:71] worker 8 working on proxies task. I0703 22:01:21.208636 1 tasks_processing.go:71] worker 10 working on machine_healthchecks task. I0703 22:01:21.207801 1 tasks_processing.go:71] worker 30 working on storage_classes task. I0703 22:01:21.207812 1 tasks_processing.go:69] worker 9 listening for tasks. I0703 22:01:21.209261 1 tasks_processing.go:71] worker 9 working on image task. I0703 22:01:21.207805 1 tasks_processing.go:71] worker 21 working on validating_webhook_configurations task. I0703 22:01:21.207809 1 tasks_processing.go:71] worker 18 working on certificate_signing_requests task. I0703 22:01:21.207818 1 tasks_processing.go:71] worker 19 working on node_logs task. I0703 22:01:21.207840 1 tasks_processing.go:71] worker 1 working on dvo_metrics task. I0703 22:01:21.207845 1 tasks_processing.go:71] worker 26 working on machine_configs task. I0703 22:01:21.208506 1 tasks_processing.go:71] worker 6 working on lokistack task. I0703 22:01:21.208509 1 tasks_processing.go:71] worker 12 working on nodenetworkstates task. I0703 22:01:21.208569 1 tasks_processing.go:71] worker 7 working on openshift_machine_api_events task. I0703 22:01:21.208575 1 tasks_processing.go:71] worker 31 working on ceph_cluster task. I0703 22:01:21.208636 1 tasks_processing.go:71] worker 27 working on pdbs task. I0703 22:01:21.208644 1 tasks_processing.go:71] worker 11 working on openstack_controlplanes task. I0703 22:01:21.208901 1 tasks_processing.go:71] worker 13 working on tsdb_status task. I0703 22:01:21.208907 1 tasks_processing.go:71] worker 25 working on ingress task. I0703 22:01:21.209151 1 tasks_processing.go:71] worker 2 working on install_plans task. I0703 22:01:21.209156 1 tasks_processing.go:71] worker 23 working on support_secret task. I0703 22:01:21.209160 1 tasks_processing.go:71] worker 3 working on overlapping_namespace_uids task. I0703 22:01:21.209164 1 tasks_processing.go:71] worker 24 working on cluster_apiserver task. I0703 22:01:21.212452 1 tasks_processing.go:71] worker 29 working on openshift_logging task. E0703 22:01:21.212469 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0703 22:01:21.212482 1 gather.go:177] gatherer "clusterconfig" function "machines" took 3.792119ms to process 0 records E0703 22:01:21.214770 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.178.144:9091: connect: connection refused I0703 22:01:21.214810 1 tasks_processing.go:71] worker 13 working on container_runtime_configs task. E0703 22:01:21.214827 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.178.144:9091: connect: connection refused I0703 22:01:21.214877 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.330813ms to process 0 records I0703 22:01:21.215073 1 tasks_processing.go:71] worker 10 working on nodenetworkconfigurationpolicies task. E0703 22:01:21.215081 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0703 22:01:21.215091 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 6.040417ms to process 0 records I0703 22:01:21.215346 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0703 22:01:21.215366 1 controller.go:326] The initial operator extension status is healthy I0703 22:01:21.215376 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0703 22:01:21.215382 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0703 22:01:21.215391 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0703 22:01:21.215411 1 controller.go:489] The operator is still being initialized I0703 22:01:21.215422 1 controller.go:512] The operator is healthy I0703 22:01:21.215544 1 tasks_processing.go:71] worker 22 working on version task. I0703 22:01:21.215596 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0703 22:01:21.215712 1 operator.go:288] started I0703 22:01:21.215829 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0703 22:01:21.215574 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0703 22:01:21.216504 1 recorder.go:75] Recording config/infrastructure with fingerprint=a06e25271c527c82a111bb40a77c151605ce4e4ab4c716e4dfe38353abe71810 I0703 22:01:21.216519 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 6.951853ms to process 1 records I0703 22:01:21.216529 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 7.242565ms to process 0 records I0703 22:01:21.216546 1 tasks_processing.go:71] worker 5 working on openstack_version task. I0703 22:01:21.216685 1 tasks_processing.go:71] worker 20 working on feature_gates task. I0703 22:01:21.216747 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=3ade7cd8377e954db837237f25c56c4088495719e0d2cd6dbcb9dd825ce8b513 I0703 22:01:21.216760 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 7.881686ms to process 1 records I0703 22:01:21.216839 1 recorder.go:75] Recording config/proxy with fingerprint=a587bf71ba3b4befc182443da41f86a5e800758940c3473d3bea73bdf730da12 I0703 22:01:21.216852 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 7.536662ms to process 1 records I0703 22:01:21.216861 1 tasks_processing.go:71] worker 8 working on mutating_webhook_configurations task. I0703 22:01:21.219436 1 tasks_processing.go:71] worker 12 working on service_accounts task. I0703 22:01:21.219456 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 9.508526ms to process 0 records I0703 22:01:21.219939 1 tasks_processing.go:71] worker 11 working on image_registries task. I0703 22:01:21.219957 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.494253ms to process 0 records I0703 22:01:21.219967 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 10.071021ms to process 0 records I0703 22:01:21.219972 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 12.046899ms to process 0 records I0703 22:01:21.219978 1 tasks_processing.go:71] worker 0 working on aggregated_monitoring_cr_names task. I0703 22:01:21.220088 1 tasks_processing.go:71] worker 6 working on openstack_dataplanenodesets task. I0703 22:01:21.226534 1 tasks_processing.go:71] worker 31 working on silenced_alerts task. I0703 22:01:21.226572 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.252338ms to process 0 records I0703 22:01:21.226584 1 gather_logs.go:145] no pods in namespace were found I0703 22:01:21.226605 1 tasks_processing.go:71] worker 14 working on machine_autoscalers task. I0703 22:01:21.226630 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 17.942427ms to process 0 records I0703 22:01:21.227219 1 tasks_processing.go:71] worker 5 working on machine_sets task. I0703 22:01:21.227344 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 10.656059ms to process 0 records I0703 22:01:21.227427 1 tasks_processing.go:71] worker 10 working on config_maps task. I0703 22:01:21.227434 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 12.340919ms to process 0 records I0703 22:01:21.227569 1 tasks_processing.go:71] worker 29 working on openstack_dataplanedeployments task. I0703 22:01:21.227825 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 15.102985ms to process 0 records I0703 22:01:21.227882 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 12.838469ms to process 0 records I0703 22:01:21.227946 1 tasks_processing.go:71] worker 13 working on machine_config_pools task. I0703 22:01:21.230098 1 tasks_processing.go:71] worker 9 working on nodes task. I0703 22:01:21.230204 1 recorder.go:75] Recording config/image with fingerprint=c759d5e04808772141a583f970ee0cc96226ab7f3a25a0bbfedb48a780fcef55 I0703 22:01:21.230223 1 gather.go:177] gatherer "clusterconfig" function "image" took 20.82198ms to process 1 records E0703 22:01:21.230240 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0703 22:01:21.230249 1 tasks_processing.go:71] worker 23 working on container_images task. I0703 22:01:21.230251 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 19.517815ms to process 0 records I0703 22:01:21.230383 1 tasks_processing.go:71] worker 17 working on schedulers task. I0703 22:01:21.230767 1 recorder.go:75] Recording config/network with fingerprint=63bacdb4416eaa58ff4a126fbb8f0105fea4c014def799c1eb2fe2a508f8df51 I0703 22:01:21.230798 1 gather.go:177] gatherer "clusterconfig" function "networks" took 22.384211ms to process 1 records I0703 22:01:21.239182 1 tasks_processing.go:71] worker 24 working on sap_datahubs task. I0703 22:01:21.239712 1 recorder.go:75] Recording config/apiserver with fingerprint=9f567876e2421dfec3f487ae9b7296d7ccc26f960ed3b524aa1f696e426120ed I0703 22:01:21.239770 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.245376ms to process 1 records I0703 22:01:21.239832 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 12.09958ms to process 0 records I0703 22:01:21.239870 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 11.717996ms to process 0 records I0703 22:01:21.239908 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 29.198865ms to process 0 records I0703 22:01:21.239934 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 12.795054ms to process 0 records I0703 22:01:21.239963 1 tasks_processing.go:71] worker 14 working on pod_network_connectivity_checks task. I0703 22:01:21.240496 1 recorder.go:75] Recording config/ingress with fingerprint=630ae261e9be7ee49ec2f7da097e018cd0ed374992a9c35dc8eb29aeeca76616 I0703 22:01:21.240572 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 29.586485ms to process 1 records I0703 22:01:21.240618 1 tasks_processing.go:71] worker 7 working on clusterroles task. I0703 22:01:21.240623 1 tasks_processing.go:71] worker 25 working on metrics task. I0703 22:01:21.241429 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=3542d1f39932019aad0a6e0d7d258d92ea873b921a48d70d654cad6a15fb304a I0703 22:01:21.241514 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=24b386f41c18d20e54c3963a9ac3dfdaa5ea540e1887c4088c54a2acf414f282 I0703 22:01:21.240628 1 tasks_processing.go:71] worker 29 working on operators_pods_and_events task. I0703 22:01:21.240631 1 tasks_processing.go:71] worker 5 working on authentication task. I0703 22:01:21.241116 1 tasks_processing.go:71] worker 8 working on active_alerts task. I0703 22:01:21.241573 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=845b25e2907383ac3508fab43c6cab5db3471c010b2c6c771da09d8460860438 I0703 22:01:21.242874 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 24.240569ms to process 3 records I0703 22:01:21.243154 1 tasks_processing.go:71] worker 11 working on sap_config task. I0703 22:01:21.243311 1 controller.go:212] Source scaController *sca.Controller is not ready I0703 22:01:21.243326 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0703 22:01:21.243331 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0703 22:01:21.243335 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0703 22:01:21.243339 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0703 22:01:21.243362 1 controller.go:489] The operator is still being initialized I0703 22:01:21.243370 1 controller.go:512] The operator is healthy I0703 22:01:21.243712 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=1391fd910f70c355f7593620c8c4f1a75349eadd7771a8520cf0ae95ecb8e011 I0703 22:01:21.243770 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 22.463371ms to process 1 records I0703 22:01:21.243943 1 tasks_processing.go:71] worker 20 working on cost_management_metrics_configs task. I0703 22:01:21.243973 1 recorder.go:75] Recording config/featuregate with fingerprint=b3215c9f7c9702f2ad6c2b0793ef120dba1be35d762042f133da0a86d3571c70 I0703 22:01:21.244138 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 26.073315ms to process 1 records I0703 22:01:21.244223 1 tasks_processing.go:71] worker 21 working on storage_cluster task. I0703 22:01:21.244273 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=576cc6bbd7a02bd7c5e57c37d785278caf8cf22e709a50e151ab12ff7077238f I0703 22:01:21.244383 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=940124449e36300756b5bc5540381529efe9b37cb3f8cbe9bb2da8b8e4d7e4f2 I0703 22:01:21.244524 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=a56c7bc425056551c5f957514452939d52df4a975c2dfe6b54c2faa8d7f90203 I0703 22:01:21.244613 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=15f3e363581186fbdd2ebbc2bd376d0df5bd669749f5fc6b9947de1750d4f4ee I0703 22:01:21.244680 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=6f916e87f44e27bcf732dbac82b79887d830043c650a7cfd2164834034f91131 I0703 22:01:21.244722 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=6c108dbccf5c1ab9e644c0d89b4b373af5fa2e340a976d3c4fa827a82d544fd1 I0703 22:01:21.244763 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=c94ae0672a3489c347ba970fc100707314d3556eba7bba78f18ddd4d9ec2aba9 I0703 22:01:21.244821 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=7765a5534d1aa543c097602f899e1ddc00aba4356609fc36e2f565a5b16b7f65 I0703 22:01:21.244863 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=524aa1e2ba9a7a27f4dc0c650c745bad7d4823cc9516539b22b654e4423a0aa1 I0703 22:01:21.244917 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=0e7ddeb016489ec49095d8c5e89689d38fa672f408b66a87d01559f82b305a60 I0703 22:01:21.244941 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=3d297361985f26c8f9afd76235d68030880c3d00df40a03d39e778bfb64f3913 I0703 22:01:21.244963 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=f0ecd04ec9d7b2374ccdbd8d64dd44ec71bc60f4010c528ea643457297f3b4cb I0703 22:01:21.244988 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=35e1eabc10da54d2837234bc0f2274e6e0d57efa0dc29ac175b2b09ecab346c5 I0703 22:01:21.244998 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 34.376373ms to process 13 records I0703 22:01:21.245013 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 24.577798ms to process 0 records I0703 22:01:21.245022 1 tasks_processing.go:71] worker 6 working on oauths task. I0703 22:01:21.245436 1 tasks_processing.go:71] worker 3 working on jaegers task. I0703 22:01:21.245455 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0703 22:01:21.245469 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 34.75095ms to process 1 records E0703 22:01:21.245625 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.178.144:9091: connect: connection refused I0703 22:01:21.245650 1 tasks_processing.go:71] worker 25 working on crds task. E0703 22:01:21.245676 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.178.144:9091: connect: connection refused I0703 22:01:21.245704 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.347237ms to process 0 records I0703 22:01:21.247923 1 tasks_processing.go:74] worker 13 stopped. I0703 22:01:21.247938 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 19.937395ms to process 0 records I0703 22:01:21.248133 1 tasks_processing.go:74] worker 30 stopped. I0703 22:01:21.248251 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=09381072e660d0d983ef95637f6a3b50e56c9a66c22ef55e5553a8b61272acef I0703 22:01:21.248378 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=5e9f11f3fa73b3ed4a566f0eeb731cb15dca9e6c4d21879bcce7174aa6a1faac I0703 22:01:21.248392 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 38.915308ms to process 2 records I0703 22:01:21.248399 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 38.744194ms to process 0 records I0703 22:01:21.248447 1 tasks_processing.go:74] worker 19 stopped. I0703 22:01:21.248471 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=932213be07af601c0d7f2f7ac41fe20d611901e8b76290c6f2fb463e358ccc0c I0703 22:01:21.248510 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 17.88185ms to process 1 records I0703 22:01:21.248476 1 tasks_processing.go:74] worker 17 stopped. I0703 22:01:21.252002 1 tasks_processing.go:74] worker 24 stopped. I0703 22:01:21.252020 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 12.75552ms to process 0 records I0703 22:01:21.252190 1 tasks_processing.go:74] worker 14 stopped. E0703 22:01:21.252235 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0703 22:01:21.252269 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 12.201309ms to process 0 records I0703 22:01:21.252665 1 tasks_processing.go:74] worker 27 stopped. I0703 22:01:21.252854 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=488a64510b6883ed4e2b005970b4f3a5f3112b82ae8eaaef79148cdf1c429e30 I0703 22:01:21.252920 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=ca4a4ef0604c81d82386ec3a2503a0cccb2171fbb2f7f57081df1f3a278424fa I0703 22:01:21.252943 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=bfb5ad5dfbf3b0c5bd0e6210107530b1ec891e1ced61da9840049c243ed39e3c I0703 22:01:21.252965 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=3afe1468f22bd316c3155edcfc608125cadbcb5d3d371cd63a1168f9b80d94be I0703 22:01:21.252990 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=fd0e2d90c8636d46f7c1c0f430cdddcede8fe8cacec1d9ca5c7f7808231b3fa7 I0703 22:01:21.253015 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=425553341d419c71a0eea50a40c3d5d795a48c92ec61b2dc5098bac0317ba78c I0703 22:01:21.253038 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=5a5e5397014e47b36685934a96e834476a7948562228440237437bd45a8c093c I0703 22:01:21.253066 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=0b74a525e4ec3914228d2c51da6e18e6336cdcfe62b35547b3d85392664f5e25 I0703 22:01:21.253114 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=b1c47e665f9ca9161bc5d6c3b475829eb3fe1447e08bf47214e95740a38928df I0703 22:01:21.253136 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=143f00e472b78ca6aa2310fadde36041daa196e610ced02bc55facad219690ab I0703 22:01:21.253164 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e41d845df8dcde761f3b63f9dbefd783204740cfb30c85a824f1794410bc9487 I0703 22:01:21.253176 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 42.330212ms to process 11 records I0703 22:01:21.253428 1 tasks_processing.go:74] worker 9 stopped. I0703 22:01:21.253899 1 recorder.go:75] Recording config/node/ip-10-0-1-217.ec2.internal with fingerprint=1b68ba0946821ef84864e54288f8f8a427047feaf4ac6a0a9d01b6fdb5cbb519 I0703 22:01:21.254000 1 recorder.go:75] Recording config/node/ip-10-0-2-97.ec2.internal with fingerprint=7229d24b3aff75a6e01dad4cab645dc918208abfb9b1ae0eaa88f25207339243 I0703 22:01:21.254013 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 23.314411ms to process 2 records I0703 22:01:21.257414 1 tasks_processing.go:74] worker 18 stopped. I0703 22:01:21.257479 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 48.051578ms to process 0 records I0703 22:01:21.262695 1 tasks_processing.go:74] worker 1 stopped. I0703 22:01:21.262719 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=108780cb24f15bf572a4031966507398a0d0c6a28b3a008952f53b481dc191a9 I0703 22:01:21.262740 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 53.102148ms to process 1 records I0703 22:01:21.266994 1 tasks_processing.go:74] worker 11 stopped. I0703 22:01:21.267011 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 23.824196ms to process 0 records I0703 22:01:21.267357 1 tasks_processing.go:74] worker 20 stopped. I0703 22:01:21.267374 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.361496ms to process 0 records I0703 22:01:21.267393 1 tasks_processing.go:74] worker 26 stopped. I0703 22:01:21.267422 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0703 22:01:21.267435 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 57.692429ms to process 1 records I0703 22:01:21.267452 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 23.170533ms to process 0 records I0703 22:01:21.267459 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 21.962678ms to process 0 records I0703 22:01:21.267466 1 tasks_processing.go:74] worker 3 stopped. I0703 22:01:21.267471 1 tasks_processing.go:74] worker 21 stopped. I0703 22:01:21.268336 1 tasks_processing.go:74] worker 15 stopped. I0703 22:01:21.268417 1 recorder.go:75] Recording config/olm_operators with fingerprint=0345b999b00538e8973039585483186087f9c452a0cadfd6e7711796b5ec5ebc I0703 22:01:21.268475 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 59.594397ms to process 1 records I0703 22:01:21.270704 1 tasks_processing.go:74] worker 7 stopped. I0703 22:01:21.271057 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=42dc35d5a4f6363d057d30a74183bc471da951309aa593be2a004c12c443b7c5 I0703 22:01:21.271215 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2d4c50c3074e677dffaac401bd8d2ef1a7164442ad7e16feb70e00e8649d770a I0703 22:01:21.271226 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 29.982688ms to process 2 records I0703 22:01:21.271315 1 tasks_processing.go:74] worker 6 stopped. I0703 22:01:21.271487 1 recorder.go:75] Recording config/oauth with fingerprint=05f49874bbd3cdb0b1bf4533b147d212436fa97c7ded7d02a85dd62b771c2305 I0703 22:01:21.271518 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 25.873692ms to process 1 records I0703 22:01:21.275760 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0703 22:01:21.275955 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0703 22:01:21.276028 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0703 22:01:21.277188 1 tasks_processing.go:74] worker 5 stopped. I0703 22:01:21.277520 1 recorder.go:75] Recording config/authentication with fingerprint=873dc82706b65df86f115d488bfec8c1628bc6194332bf7b5bbdca453251cf48 I0703 22:01:21.277566 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 34.517645ms to process 1 records I0703 22:01:21.277905 1 tasks_processing.go:74] worker 0 stopped. I0703 22:01:21.277995 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 57.91776ms to process 0 records I0703 22:01:21.278068 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 71.249693ms to process 0 records I0703 22:01:21.278084 1 tasks_processing.go:74] worker 4 stopped. I0703 22:01:21.279542 1 base_controller.go:82] Caches are synced for ConfigController I0703 22:01:21.279584 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0703 22:01:21.281462 1 tasks_processing.go:74] worker 23 stopped. I0703 22:01:21.282274 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0703 22:01:21.283201 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-lkd9t with fingerprint=48641cb6e42fd5e46878d99b484bfe3d983a64c37e250e4c0c498879558dce89 I0703 22:01:21.283320 1 recorder.go:75] Recording config/pod/openshift-monitoring/cluster-monitoring-operator-698978dbbf-7kj8p with fingerprint=6bd692287c258ed69aa4fed1d2f491df3a0cdbd652196fa6a3eaddc8b486ad78 I0703 22:01:21.283583 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-5lmtd with fingerprint=61bd3063563bdfe5153596a134a34e23c40612a111144d633c0d7e965a391d18 I0703 22:01:21.283894 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-gcp2n with fingerprint=79f7180f8cfb5b78ed119b48005a749868ca008694e29be9b271c154f8f4819c I0703 22:01:21.283967 1 recorder.go:75] Recording config/running_containers with fingerprint=35b9f4f5bdd46b04795eef7e0509f5ba4ef7158e412a9a89d51b83eb2a31713d I0703 22:01:21.283976 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 51.192556ms to process 5 records I0703 22:01:21.284082 1 tasks_processing.go:74] worker 25 stopped. I0703 22:01:21.284519 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=6c13b1eb1445505e5ce8e3ad7a62a430ae5747270bb4b8dcca3a3e69f251f32f I0703 22:01:21.284730 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=cc949da441c09ae61d0f7ca34ca7d63ef978ca504ab18aebeafcc67c4bc676be I0703 22:01:21.284740 1 gather.go:177] gatherer "clusterconfig" function "crds" took 37.337792ms to process 2 records I0703 22:01:21.288159 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:01:21.290108 1 tasks_processing.go:74] worker 10 stopped. E0703 22:01:21.290123 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0703 22:01:21.290131 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0703 22:01:21.290136 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0703 22:01:21.290147 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0703 22:01:21.290199 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0703 22:01:21.290210 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0703 22:01:21.290216 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0703 22:01:21.290231 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0703 22:01:21.290278 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0703 22:01:21.290289 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0703 22:01:21.290297 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 62.628031ms to process 7 records I0703 22:01:21.317440 1 base_controller.go:82] Caches are synced for LoggingSyncer I0703 22:01:21.317466 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0703 22:01:21.317868 1 tasks_processing.go:74] worker 22 stopped. I0703 22:01:21.318176 1 recorder.go:75] Recording config/version with fingerprint=646ff69ac21b8ca2d1b3f3bf041f17dc78a958c4dc9f6c9409f31a99a187246d I0703 22:01:21.318189 1 recorder.go:75] Recording config/id with fingerprint=c555e7fb0179cd33c58984dea5b4ce9aba52d53d3559e1aecca22bc7e52ce95e I0703 22:01:21.318195 1 gather.go:177] gatherer "clusterconfig" function "version" took 102.271138ms to process 2 records I0703 22:01:21.327104 1 gather_cluster_operator_pods_and_events.go:121] Found 0 pods with 0 containers I0703 22:01:21.327121 1 tasks_processing.go:74] worker 29 stopped. I0703 22:01:21.327129 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 84.549037ms to process 0 records I0703 22:01:21.330452 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0703 22:01:21.334717 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0703 22:01:21.350387 1 cluster_transfer.go:102] no available accepted cluster transfer I0703 22:01:21.350400 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0703 22:01:21.352416 1 tasks_processing.go:74] worker 16 stopped. E0703 22:01:21.352429 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0703 22:01:21.352436 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ratrkdkluad89chjacv8i03daia4uc2-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ratrkdkluad89chjacv8i03daia4uc2-primary-cert-bundle-secret" not found I0703 22:01:21.352489 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=a3f7660a3e49afd952aa62fb7fd24b4ca562e96382877a8787ca22e507763744 I0703 22:01:21.352500 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 144.578863ms to process 1 records I0703 22:01:21.466301 1 sca.go:266] etc-pki-entitlement secret successfully created I0703 22:01:21.466314 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0703 22:01:22.278928 1 tasks_processing.go:74] worker 31 stopped. I0703 22:01:22.278947 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0703 22:01:22.278958 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.052371158s to process 1 records I0703 22:01:22.278988 1 recorder.go:75] Recording config/alerts with fingerprint=9bb09213ce5f92f11fbe0f5f904bccf5911ff6475d41394bd716a73a13521080 I0703 22:01:22.278994 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.036136389s to process 1 records I0703 22:01:22.278999 1 tasks_processing.go:74] worker 8 stopped. I0703 22:01:23.686700 1 tasks_processing.go:74] worker 28 stopped. I0703 22:01:23.686887 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=ae4a71b9a350ca14cbb564d45f67a54a8e2f638922b23b8af999172e4b433065 I0703 22:01:23.686958 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0703 22:01:23.686999 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=dd0d55b05bb7eb7b95418a9e41aac85591a3efb50da3eb82f3d98351a95d2bc0 I0703 22:01:23.687015 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0703 22:01:23.687052 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=9e2b8207fe1e65c2dd069cfeb3d8e7bdda81864b8167c31354c35def3b9dab9a I0703 22:01:23.687075 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0703 22:01:23.687112 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=de0227b0435b999b2adab3c2094fa2ccd53d56ef177fea67695476ad24d90fa0 I0703 22:01:23.687158 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=5d33fa048af9b65d65b97ad474c9fb9d1f49cecdaaba8b04fea3e2014e3666b7 I0703 22:01:23.687196 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=90fb1006b3454c585db02c2502732177ad2bc77dfaade776c91f107964a5c64e I0703 22:01:23.687247 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=0f11842ec3fe8e66453d1a3488ac15b1f43a8897f996e93d1083916802447460 I0703 22:01:23.687667 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=c6411af1b9b0b5b88f175107a2e86143f62e0c726c6b17e3892d9bc07a2db93a I0703 22:01:23.687715 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=e49549b2c7274d7d537126f200652a7d52affbed84d07f3ffee42206b85a4345 I0703 22:01:23.687738 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0703 22:01:23.687771 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=64fc286dcfd3fa7f7eb4ac1a3a1f71f3a2b689d8a87ef3eb54799a30fe03eab2 I0703 22:01:23.687808 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0703 22:01:23.687842 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=625c65e62162139a2923b33750e37071fe97e8aa243180ae25e4334cea123ab8 I0703 22:01:23.687856 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0703 22:01:23.687889 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=9e2d36ba7e1f1d5e7639694f2df1b1336b2e4a320ee9ae39b1a7272eac9b4447 I0703 22:01:23.687903 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0703 22:01:23.687939 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=ae950466313eb4499ee82403c53752b52a95749f2da926616a9667e0c0ad1813 I0703 22:01:23.688115 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=2f5943d90156d63f5de3f328338324600743309512057992088356296004d57b I0703 22:01:23.688131 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0703 22:01:23.688145 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0703 22:01:23.688190 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0703 22:01:23.688228 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=f26b26d5fbf4e134482105a6e2cda6bbdfb65a8ac90f2acd2dd3ae7fd8472760 I0703 22:01:23.688272 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=ede1723149bb6219aeff51c14fa89b8a94dd21dd0c661fcf03c2c9f585976878 I0703 22:01:23.688286 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0703 22:01:23.688312 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=018fef1ebff04f2c3771a1fe6ae13d85e75c3aa8bf6e10752958961cb2d7f3a0 I0703 22:01:23.688330 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0703 22:01:23.688367 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=701b085ec6fc4a53488454ea0dd90dc76c51750cad5e8c47fbd7db553d01d3d7 I0703 22:01:23.688381 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0703 22:01:23.688403 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=f651bfa5d071d1971fba44157c36e7ace8822c0da45a8ea8a23058d0b5dbb3b4 I0703 22:01:23.688429 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=d6dda5fe108230b1853c956b681f145c6cbf7ca71da3b4d3bd84b75bbdeab630 I0703 22:01:23.688454 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=49d37ad0cf081dec7f29442a489f59c8bca3bc288ba82d77b68eb3635e19f997 I0703 22:01:23.688491 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=ed55db74f81c3146595cc5e77360f5d536cb4e75e72cb50caab493cb7e1c0778 I0703 22:01:23.688505 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0703 22:01:23.688555 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b87c0c8c73ec9f60bbefe5ddfaf716e12ffaf6f158a6b12499f6363147ad31da I0703 22:01:23.688581 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0703 22:01:23.688594 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0703 22:01:23.688605 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.478019661s to process 39 records I0703 22:01:34.251164 1 tasks_processing.go:74] worker 2 stopped. I0703 22:01:34.251206 1 recorder.go:75] Recording config/installplans with fingerprint=e3c18441424aaa81f54ea9723d194eacb5824eb47178047bd893de11825bc8a2 I0703 22:01:34.251219 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.040575103s to process 1 records I0703 22:01:35.025964 1 tasks_processing.go:74] worker 12 stopped. I0703 22:01:35.026246 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=ae29e7659fb4cd0f04bcc181913d3f8b69c20a612f3ed6bf2f2479314a8cc58f I0703 22:01:35.026265 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.80650183s to process 1 records E0703 22:01:35.026322 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.82s with: function \"machines\" failed with an error, function \"tsdb_status\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"metrics\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error" I0703 22:01:35.027424 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "tsdb_status" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "metrics" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error I0703 22:01:35.027437 1 periodic.go:209] Running workloads gatherer I0703 22:01:35.027453 1 tasks_processing.go:45] number of workers: 2 I0703 22:01:35.027460 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 22:01:35.027465 1 tasks_processing.go:71] worker 1 working on workload_info task. I0703 22:01:35.027471 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 22:01:35.027490 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0703 22:01:35.033646 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-97.ec2.internal... I0703 22:01:35.033658 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-217.ec2.internal... I0703 22:01:35.052656 1 tasks_processing.go:74] worker 0 stopped. I0703 22:01:35.052671 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.14364ms to process 0 records I0703 22:01:40.976351 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 6s I0703 22:01:40.999883 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 23s for image data I0703 22:01:41.008856 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0703 22:01:41.020351 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (11ms) I0703 22:01:41.029494 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0703 22:01:41.038342 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (9ms) I0703 22:01:41.047138 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (9ms) I0703 22:01:41.055428 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (8ms) I0703 22:01:41.063475 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (8ms) I0703 22:01:41.071323 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (8ms) I0703 22:01:41.079499 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (8ms) I0703 22:01:41.087199 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (8ms) I0703 22:01:41.108375 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (21ms) I0703 22:01:41.209040 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (101ms) I0703 22:01:41.309210 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0703 22:01:41.411755 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (103ms) I0703 22:01:41.508541 1 gather_workloads_info.go:387] No image sha256:88437b35173fe2e8a4d9f148ec17c754bee4df3f723b3a5676955eef4f363d31 (97ms) I0703 22:01:41.609629 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (101ms) I0703 22:01:41.708293 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (99ms) I0703 22:01:41.809369 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0703 22:01:41.908943 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0703 22:01:42.009065 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0703 22:01:42.108651 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0703 22:01:42.208563 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (100ms) I0703 22:01:42.308388 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0703 22:01:42.410806 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (102ms) I0703 22:01:42.508497 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (98ms) I0703 22:01:42.609521 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (101ms) I0703 22:01:42.708439 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0703 22:01:42.809288 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (101ms) I0703 22:01:42.908977 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0703 22:01:43.019249 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (110ms) I0703 22:01:43.118056 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (99ms) I0703 22:01:43.210001 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (92ms) I0703 22:01:43.308643 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (99ms) I0703 22:01:43.409643 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (101ms) I0703 22:01:43.509486 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0703 22:01:43.609639 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0703 22:01:43.708218 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (98ms) I0703 22:01:43.808961 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (101ms) I0703 22:01:43.908639 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (100ms) I0703 22:01:44.010364 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (102ms) I0703 22:01:44.108649 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (98ms) I0703 22:01:44.208600 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0703 22:01:44.308500 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0703 22:01:44.408406 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0703 22:01:44.508207 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0703 22:01:44.608512 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0703 22:01:44.708364 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0703 22:01:44.809008 1 gather_workloads_info.go:387] No image sha256:09e27b3a67ebeda04d3e597e0561bfb40550de78bdecb53f2f9ff3dfb1cfe7ea (101ms) I0703 22:01:44.908523 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (99ms) I0703 22:01:45.010883 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (102ms) I0703 22:01:45.010920 1 tasks_processing.go:74] worker 1 stopped. I0703 22:01:45.011212 1 recorder.go:75] Recording config/workload_info with fingerprint=2c12335cff1aba6af878a932be7bdfaea25f7159b60ae8caf76c80c3116997ec I0703 22:01:45.011231 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.983446617s to process 1 records I0703 22:01:45.011257 1 periodic.go:256] Periodic gather workloads completed in 9.983s I0703 22:01:45.011267 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0703 22:01:45.011273 1 periodic.go:209] Running conditional gatherer I0703 22:01:45.020625 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0703 22:01:45.124729 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0703 22:01:45.127082 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.178.144:9091: connect: connection refused I0703 22:01:45.127153 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0703 22:01:45.133915 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0703 22:01:45.133930 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133936 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133941 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133946 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133951 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133956 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133959 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133963 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133965 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0703 22:01:45.133969 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0703 22:01:45.133983 1 tasks_processing.go:45] number of workers: 3 I0703 22:01:45.133993 1 tasks_processing.go:69] worker 2 listening for tasks. I0703 22:01:45.133997 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0703 22:01:45.133996 1 tasks_processing.go:69] worker 0 listening for tasks. I0703 22:01:45.134010 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0703 22:01:45.134015 1 tasks_processing.go:69] worker 1 listening for tasks. I0703 22:01:45.134024 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0703 22:01:45.134039 1 tasks_processing.go:74] worker 1 stopped. I0703 22:01:45.134049 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0703 22:01:45.134059 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 796ns to process 1 records I0703 22:01:45.134126 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0703 22:01:45.134136 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 943ns to process 1 records I0703 22:01:45.134142 1 tasks_processing.go:74] worker 2 stopped. I0703 22:01:45.134243 1 tasks_processing.go:74] worker 0 stopped. I0703 22:01:45.134255 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 227.715µs to process 0 records I0703 22:01:45.134270 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0703 22:01:45.134288 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0703 22:01:45.159880 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=dd92fe65510c05b84170ea8b780ddd453427e4b9adaa33c0e5ba2ff3450c01cb I0703 22:01:45.160043 1 diskrecorder.go:70] Writing 113 records to /var/lib/insights-operator/insights-2026-07-03-220145.tar.gz I0703 22:01:45.167590 1 diskrecorder.go:51] Wrote 113 records to disk in 7ms I0703 22:01:45.167631 1 periodic.go:278] Gathering cluster info every 2h0m0s I0703 22:01:45.167651 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0703 22:02:51.206751 1 diskrecorder.go:223] Found files to send: insights-2026-07-03-220145.tar.gz I0703 22:02:51.206796 1 insightsuploader.go:150] Checking archives to upload periodically every 15m23.232604711s I0703 22:02:51.206808 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0703 22:02:51.217416 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0703 22:02:51.331996 1 requests.go:87] Successfully reported id=2026-07-03T22:02:51Z x-rh-insights-request-id=3d91a0864cff473b8da60353b1b507e7, wrote=55909 I0703 22:02:51.332051 1 insightsuploader.go:187] Uploaded report successfully in 125.237725ms I0703 22:02:51.332072 1 controller.go:128] Initializing last reported time to 2026-07-03T22:02:51Z I0703 22:02:51.332101 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0703 22:02:51.332111 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0703 22:02:51.332119 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0703 22:02:51.338311 1 controller.go:512] The operator is healthy I0703 22:03:21.223132 1 controller.go:512] The operator is healthy I0703 22:03:56.892767 1 insightsreport.go:137] Pulling report from smart-proxy I0703 22:03:56.892812 1 insightsreport.go:149] Retrieving report I0703 22:03:56.900547 1 requests.go:110] Retrieving report for cluster: c7204458-122f-48a3-accc-aad18925e513 I0703 22:03:56.900561 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/c7204458-122f-48a3-accc-aad18925e513/reports I0703 22:03:56.903339 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/c7204458-122f-48a3-accc-aad18925e513/reports I0703 22:03:58.003034 1 insightsreport.go:184] Report retrieved I0703 22:03:58.041908 1 insightsreport.go:239] Report retrieved correctly I0703 22:05:21.223223 1 controller.go:512] The operator is healthy I0703 22:06:21.198579 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:06:21.204282 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:06:21.204302 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:06:21.208507 1 secretconfigobserver.go:119] support secret does not exist I0703 22:07:21.224422 1 controller.go:512] The operator is healthy I0703 22:09:21.221807 1 controller.go:512] The operator is healthy I0703 22:11:21.193615 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:11:21.197936 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:11:21.209018 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:11:21.214543 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:11:21.214565 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:11:21.219318 1 secretconfigobserver.go:119] support secret does not exist I0703 22:11:21.222180 1 controller.go:512] The operator is healthy I0703 22:13:21.222261 1 controller.go:512] The operator is healthy I0703 22:15:21.223392 1 controller.go:512] The operator is healthy I0703 22:16:21.220230 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:16:21.231031 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:16:21.231056 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:16:21.235362 1 secretconfigobserver.go:119] support secret does not exist I0703 22:17:21.223704 1 controller.go:512] The operator is healthy I0703 22:18:19.566391 1 insightsuploader.go:146] Nothing to report since 2026-07-03T22:02:51Z I0703 22:19:21.222642 1 controller.go:512] The operator is healthy I0703 22:21:21.195799 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:21:21.200305 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:21:21.223183 1 controller.go:512] The operator is healthy I0703 22:21:21.236239 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:21:21.241514 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:21:21.241530 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:21:21.245925 1 secretconfigobserver.go:119] support secret does not exist I0703 22:23:21.223439 1 controller.go:512] The operator is healthy I0703 22:25:21.222479 1 controller.go:512] The operator is healthy I0703 22:26:21.246387 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:26:21.251756 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:26:21.251774 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:26:21.255627 1 secretconfigobserver.go:119] support secret does not exist I0703 22:27:21.223807 1 controller.go:512] The operator is healthy I0703 22:29:21.226167 1 controller.go:512] The operator is healthy I0703 22:31:21.196131 1 configmapobserver.go:84] configmaps "insights-config" not found I0703 22:31:21.222863 1 controller.go:512] The operator is healthy I0703 22:31:21.255933 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:31:21.263093 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:31:21.263109 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:31:21.267213 1 secretconfigobserver.go:119] support secret does not exist I0703 22:33:21.226153 1 controller.go:512] The operator is healthy I0703 22:33:32.676269 1 insightsuploader.go:146] Nothing to report since 2026-07-03T22:02:51Z I0703 22:35:21.224649 1 controller.go:512] The operator is healthy I0703 22:36:21.267955 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0703 22:36:21.273639 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0703 22:36:21.273660 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0703 22:36:21.277750 1 secretconfigobserver.go:119] support secret does not exist I0703 22:37:21.226564 1 controller.go:512] The operator is healthy