W0705 14:15:46.078613 1 cmd.go:257] Using insecure, self-signed certificates I0705 14:15:46.339395 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:15:46.339745 1 observer_polling.go:159] Starting file observer I0705 14:15:46.613277 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0705 14:15:46.613470 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0705 14:15:46.614024 1 secure_serving.go:57] Forcing use of http/1.1 only W0705 14:15:46.614044 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0705 14:15:46.614048 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0705 14:15:46.614052 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0705 14:15:46.614055 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0705 14:15:46.614057 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0705 14:15:46.614059 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0705 14:15:46.614093 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0705 14:15:46.618155 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0705 14:15:46.618158 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"f97a831b-141d-4b35-8c80-651395f9126f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0705 14:15:46.622477 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0705 14:15:46.622503 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0705 14:15:46.622503 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0705 14:15:46.622516 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0705 14:15:46.622513 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0705 14:15:46.622530 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0705 14:15:46.622825 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-746208174/tls.crt::/tmp/serving-cert-746208174/tls.key" I0705 14:15:46.623128 1 secure_serving.go:213] Serving securely on [::]:8443 I0705 14:15:46.623164 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0705 14:15:46.627104 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0705 14:15:46.627127 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0705 14:15:46.627229 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0705 14:15:46.636081 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0705 14:15:46.636142 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0705 14:15:46.642965 1 secretconfigobserver.go:119] support secret does not exist I0705 14:15:46.648054 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0705 14:15:46.655360 1 secretconfigobserver.go:119] support secret does not exist I0705 14:15:46.656755 1 recorder.go:161] Pruning old reports every 4h21m25s, max age is 288h0m0s I0705 14:15:46.662174 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0705 14:15:46.662195 1 periodic.go:209] Running clusterconfig gatherer I0705 14:15:46.662197 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0705 14:15:46.662195 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0705 14:15:46.662210 1 insightsreport.go:296] Starting report retriever I0705 14:15:46.662217 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0705 14:15:46.662245 1 tasks_processing.go:45] number of workers: 32 I0705 14:15:46.662272 1 tasks_processing.go:69] worker 0 listening for tasks. I0705 14:15:46.662282 1 tasks_processing.go:71] worker 0 working on mutating_webhook_configurations task. I0705 14:15:46.662287 1 tasks_processing.go:69] worker 8 listening for tasks. I0705 14:15:46.662291 1 tasks_processing.go:69] worker 9 listening for tasks. I0705 14:15:46.662299 1 tasks_processing.go:69] worker 5 listening for tasks. I0705 14:15:46.662297 1 tasks_processing.go:69] worker 4 listening for tasks. I0705 14:15:46.662305 1 tasks_processing.go:69] worker 2 listening for tasks. I0705 14:15:46.662308 1 tasks_processing.go:69] worker 3 listening for tasks. I0705 14:15:46.662310 1 tasks_processing.go:69] worker 11 listening for tasks. I0705 14:15:46.662313 1 tasks_processing.go:69] worker 6 listening for tasks. I0705 14:15:46.662303 1 tasks_processing.go:69] worker 10 listening for tasks. I0705 14:15:46.662332 1 tasks_processing.go:69] worker 26 listening for tasks. I0705 14:15:46.662334 1 tasks_processing.go:71] worker 10 working on qemu_kubevirt_launcher_logs task. I0705 14:15:46.662341 1 tasks_processing.go:69] worker 22 listening for tasks. I0705 14:15:46.662341 1 tasks_processing.go:71] worker 6 working on pdbs task. I0705 14:15:46.662340 1 tasks_processing.go:69] worker 16 listening for tasks. I0705 14:15:46.662352 1 tasks_processing.go:69] worker 27 listening for tasks. I0705 14:15:46.662356 1 tasks_processing.go:69] worker 7 listening for tasks. I0705 14:15:46.662357 1 tasks_processing.go:71] worker 16 working on monitoring_persistent_volumes task. I0705 14:15:46.662363 1 tasks_processing.go:69] worker 28 listening for tasks. I0705 14:15:46.662365 1 tasks_processing.go:71] worker 7 working on pod_network_connectivity_checks task. I0705 14:15:46.662370 1 tasks_processing.go:71] worker 28 working on openstack_dataplanenodesets task. I0705 14:15:46.662401 1 tasks_processing.go:69] worker 23 listening for tasks. I0705 14:15:46.662415 1 tasks_processing.go:69] worker 24 listening for tasks. I0705 14:15:46.662423 1 tasks_processing.go:69] worker 25 listening for tasks. I0705 14:15:46.662431 1 tasks_processing.go:69] worker 14 listening for tasks. I0705 14:15:46.662439 1 tasks_processing.go:69] worker 12 listening for tasks. I0705 14:15:46.662451 1 tasks_processing.go:69] worker 13 listening for tasks. I0705 14:15:46.662459 1 tasks_processing.go:69] worker 18 listening for tasks. I0705 14:15:46.662467 1 tasks_processing.go:69] worker 17 listening for tasks. I0705 14:15:46.662349 1 tasks_processing.go:71] worker 22 working on infrastructures task. I0705 14:15:46.662617 1 tasks_processing.go:69] worker 29 listening for tasks. I0705 14:15:46.662631 1 tasks_processing.go:69] worker 30 listening for tasks. I0705 14:15:46.662636 1 tasks_processing.go:69] worker 15 listening for tasks. I0705 14:15:46.662642 1 tasks_processing.go:71] worker 30 working on overlapping_namespace_uids task. I0705 14:15:46.662643 1 tasks_processing.go:71] worker 15 working on clusterroles task. I0705 14:15:46.662713 1 tasks_processing.go:71] worker 8 working on tsdb_status task. W0705 14:15:46.662775 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:15:46.662796 1 tasks_processing.go:69] worker 19 listening for tasks. I0705 14:15:46.662843 1 tasks_processing.go:71] worker 18 working on machine_autoscalers task. I0705 14:15:46.662866 1 tasks_processing.go:71] worker 17 working on cost_management_metrics_configs task. I0705 14:15:46.662914 1 tasks_processing.go:69] worker 1 listening for tasks. I0705 14:15:46.662960 1 tasks_processing.go:69] worker 21 listening for tasks. I0705 14:15:46.662984 1 tasks_processing.go:71] worker 21 working on olm_operators task. I0705 14:15:46.663022 1 tasks_processing.go:71] worker 19 working on active_alerts task. I0705 14:15:46.662282 1 tasks_processing.go:69] worker 31 listening for tasks. W0705 14:15:46.663086 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:15:46.663092 1 tasks_processing.go:71] worker 31 working on aggregated_monitoring_cr_names task. I0705 14:15:46.663040 1 tasks_processing.go:71] worker 1 working on operators_pods_and_events task. I0705 14:15:46.662777 1 tasks_processing.go:71] worker 26 working on container_images task. I0705 14:15:46.662782 1 tasks_processing.go:71] worker 11 working on storage_cluster task. I0705 14:15:46.662792 1 tasks_processing.go:71] worker 27 working on networks task. I0705 14:15:46.662802 1 tasks_processing.go:71] worker 12 working on machines task. I0705 14:15:46.662807 1 tasks_processing.go:71] worker 23 working on install_plans task. I0705 14:15:46.662810 1 tasks_processing.go:71] worker 24 working on lokistack task. I0705 14:15:46.662811 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 44.754µs to process 0 records I0705 14:15:46.662814 1 tasks_processing.go:71] worker 25 working on validating_webhook_configurations task. I0705 14:15:46.663697 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 26.713µs to process 0 records I0705 14:15:46.663710 1 tasks_processing.go:71] worker 19 working on dvo_metrics task. I0705 14:15:46.662817 1 tasks_processing.go:71] worker 9 working on version task. I0705 14:15:46.662822 1 tasks_processing.go:71] worker 5 working on proxies task. I0705 14:15:46.662818 1 tasks_processing.go:71] worker 14 working on metrics task. I0705 14:15:46.662826 1 tasks_processing.go:71] worker 4 working on openshift_machine_api_events task. W0705 14:15:46.664013 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:15:46.664026 1 tasks_processing.go:71] worker 14 working on openstack_version task. I0705 14:15:46.664056 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 31.435µs to process 0 records I0705 14:15:46.662830 1 tasks_processing.go:71] worker 2 working on machine_healthchecks task. I0705 14:15:46.662834 1 tasks_processing.go:71] worker 3 working on openstack_controlplanes task. I0705 14:15:46.662831 1 tasks_processing.go:69] worker 20 listening for tasks. I0705 14:15:46.664415 1 tasks_processing.go:71] worker 20 working on ceph_cluster task. I0705 14:15:46.662855 1 tasks_processing.go:71] worker 13 working on storage_classes task. I0705 14:15:46.662867 1 tasks_processing.go:71] worker 29 working on cluster_apiserver task. I0705 14:15:46.663050 1 tasks_processing.go:71] worker 8 working on operators task. I0705 14:15:46.665627 1 tasks_processing.go:71] worker 7 working on image task. E0705 14:15:46.665673 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0705 14:15:46.665696 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 3.254174ms to process 0 records I0705 14:15:46.665701 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 3.312961ms to process 0 records I0705 14:15:46.665752 1 tasks_processing.go:71] worker 28 working on sap_datahubs task. I0705 14:15:46.669151 1 tasks_processing.go:71] worker 18 working on machine_configs task. I0705 14:15:46.669153 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 6.271182ms to process 0 records I0705 14:15:46.669168 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 5.531353ms to process 0 records I0705 14:15:46.669238 1 tasks_processing.go:71] worker 24 working on feature_gates task. I0705 14:15:46.669321 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0705 14:15:46.669376 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0705 14:15:46.669385 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0705 14:15:46.669389 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0705 14:15:46.669407 1 controller.go:489] The operator is still being initialized I0705 14:15:46.669418 1 controller.go:512] The operator is healthy I0705 14:15:46.676542 1 tasks_processing.go:71] worker 0 working on service_accounts task. I0705 14:15:46.676611 1 gather_logs.go:145] no pods in namespace were found I0705 14:15:46.676858 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=be2413903650858d36929a1c5817d286454171e4a7d2a0fd09dcf78cc1ea3863 I0705 14:15:46.676912 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=6829e4580e27df61928c82a8f31b6ce5b13900d6116c1fe2c7d7ac445aa22b14 I0705 14:15:46.676963 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=07576d0de7ead4f4338a91e757dbba5a4840bf56f62b66412cb9592e717b637e I0705 14:15:46.676977 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 14.249542ms to process 3 records I0705 14:15:46.676989 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 13.692387ms to process 0 records I0705 14:15:46.676996 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 13.339548ms to process 0 records I0705 14:15:46.677003 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 14.315366ms to process 0 records I0705 14:15:46.677008 1 tasks_processing.go:71] worker 17 working on schedulers task. I0705 14:15:46.677022 1 tasks_processing.go:71] worker 10 working on ingress task. I0705 14:15:46.677046 1 tasks_processing.go:71] worker 16 working on openshift_logging task. I0705 14:15:46.677055 1 tasks_processing.go:71] worker 11 working on silenced_alerts task. W0705 14:15:46.677080 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:15:46.677008 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 14.323582ms to process 0 records I0705 14:15:46.677219 1 tasks_processing.go:71] worker 22 working on certificate_signing_requests task. I0705 14:15:46.678998 1 recorder.go:75] Recording config/infrastructure with fingerprint=f1065045d1eae1f878dc6ee7d82ed7be8b1994b13fd9ce2c31970ab2218a4e20 I0705 14:15:46.679054 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 14.274736ms to process 1 records E0705 14:15:46.679073 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0705 14:15:46.679087 1 gather.go:177] gatherer "clusterconfig" function "machines" took 13.580005ms to process 0 records I0705 14:15:46.679110 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 31.494µs to process 0 records I0705 14:15:46.679172 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 13.346024ms to process 0 records I0705 14:15:46.679182 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 13.7257ms to process 0 records I0705 14:15:46.679385 1 tasks_processing.go:71] worker 11 working on nodenetworkconfigurationpolicies task. I0705 14:15:46.679577 1 tasks_processing.go:71] worker 4 working on machine_config_pools task. I0705 14:15:46.679732 1 tasks_processing.go:71] worker 14 working on sap_pods task. I0705 14:15:46.679592 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=5828d91a2895a3ae15699e289fa1cb7b4378384d860063cb89ed6c07865a423c I0705 14:15:46.680147 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=cc7f8ead506dd740097044122e04767cfdba8cd0ac6e7e7db250d3725edc9018 I0705 14:15:46.680230 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=631b1e353448bfdcd0a32ecea31a60e670be6240a47637e4e1c6db64bc368bf2 I0705 14:15:46.680280 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 15.86638ms to process 3 records I0705 14:15:46.679603 1 tasks_processing.go:71] worker 6 working on jaegers task. I0705 14:15:46.680523 1 tasks_processing.go:71] worker 28 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0705 14:15:46.680770 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 14.752503ms to process 0 records I0705 14:15:46.679610 1 tasks_processing.go:71] worker 12 working on config_maps task. I0705 14:15:46.681452 1 tasks_processing.go:71] worker 27 working on node_logs task. I0705 14:15:46.682182 1 recorder.go:75] Recording config/network with fingerprint=56be3fd222947f8b9809914bd6d719d55736a4002afd972314ea71e1cfb4e278 I0705 14:15:46.682206 1 gather.go:177] gatherer "clusterconfig" function "networks" took 18.105313ms to process 1 records E0705 14:15:46.682217 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0705 14:15:46.682227 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 17.422336ms to process 0 records I0705 14:15:46.682237 1 tasks_processing.go:71] worker 2 working on nodenetworkstates task. I0705 14:15:46.682689 1 tasks_processing.go:71] worker 30 working on openstack_dataplanedeployments task. I0705 14:15:46.682723 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0705 14:15:46.682838 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 20.033887ms to process 1 records I0705 14:15:46.688470 1 tasks_processing.go:71] worker 20 working on sap_config task. I0705 14:15:46.688572 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 24.04279ms to process 0 records I0705 14:15:46.688695 1 tasks_processing.go:71] worker 5 working on image_registries task. I0705 14:15:46.688820 1 recorder.go:75] Recording config/proxy with fingerprint=a635a33cf7ff8ebfcafecc4df32770987b3b1aa10190ea5c7ad7a92f1b94ed33 I0705 14:15:46.688884 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 24.839093ms to process 1 records I0705 14:15:46.689074 1 tasks_processing.go:71] worker 29 working on support_secret task. I0705 14:15:46.689077 1 recorder.go:75] Recording config/apiserver with fingerprint=0215e774eb0f375e218c6374a090d50bf980ca6a356f8266e8944830bfe9e868 I0705 14:15:46.689300 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 24.072486ms to process 1 records I0705 14:15:46.689310 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 24.48828ms to process 0 records I0705 14:15:46.689320 1 tasks_processing.go:71] worker 3 working on nodes task. I0705 14:15:46.689879 1 tasks_processing.go:71] worker 25 working on image_pruners task. I0705 14:15:46.690092 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=be14713b1838035cf3d5128cff0003814b4448c571ea1d97bb6be922ba508837 I0705 14:15:46.690275 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=fafd2819927d6690d17f34f2f5d838a83b3141bc1863cff3210045bdcc632ef7 I0705 14:15:46.690311 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=094e0a5d1506f430185467b2a5b76f69e7e3f33daf280fe2a9fdbe9225a17764 I0705 14:15:46.690352 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=935792b246ad5d35739174e30f0cc76641ee6dd2623155b780e98fbe39acee6d I0705 14:15:46.690392 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=61b5bbd5fb0c64463fcee79c8c25e22204af9562f18fc377f81c040c9bb50e56 I0705 14:15:46.690427 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=8bd98fd09da3269df194e24198d87a1f91179de4a015583fdea5fa597bfdd565 I0705 14:15:46.690459 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=f447ed4782b460ee0645b368302058da83a4c1eaa589423a64ee7067199c7846 I0705 14:15:46.690512 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=c6626f2607dbcc18da3a23b27b4e2adbf57debb5ceed20b915b8cfaa46deb065 I0705 14:15:46.690581 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=8527e82a4dbdd1edd3ac636068b9a885d3e51e0e0c87e25c78e9da83a0c707a9 I0705 14:15:46.690625 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=9e50917f17e95ccac554e2c6b24c2efe3f9970bb39c49d6f398f30c431dcad37 I0705 14:15:46.690701 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=8b25cf396fcc6cfc9b8dc26eb1b5b6a2fe4a617ba3bef1a08e89d1d993708035 I0705 14:15:46.690720 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 26.175285ms to process 11 records I0705 14:15:46.694666 1 tasks_processing.go:71] worker 11 working on ingress_certificates task. I0705 14:15:46.694683 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 15.255806ms to process 0 records I0705 14:15:46.694842 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 17.717921ms to process 0 records I0705 14:15:46.694853 1 tasks_processing.go:71] worker 16 working on authentication task. I0705 14:15:46.694955 1 tasks_processing.go:71] worker 6 working on oauths task. I0705 14:15:46.695032 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 14.587474ms to process 0 records W0705 14:15:46.696675 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0705 14:15:46.697824 1 tasks_processing.go:71] worker 13 working on machine_sets task. I0705 14:15:46.697969 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=9f56b7f313c41ef46a4dc73832fdf569dffd10230b6dcdbf408ad74eda2dda8f I0705 14:15:46.697998 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=26a60ae3b290c6d76de82772582db21f5adce30cbb554a140da57f96ba98ab4c I0705 14:15:46.698009 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 33.386047ms to process 2 records I0705 14:15:46.698019 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 15.682525ms to process 0 records I0705 14:15:46.698026 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 18.165519ms to process 0 records I0705 14:15:46.698034 1 tasks_processing.go:71] worker 14 working on crds task. I0705 14:15:46.698047 1 tasks_processing.go:71] worker 2 working on container_runtime_configs task. I0705 14:15:46.698131 1 tasks_processing.go:74] worker 30 stopped. I0705 14:15:46.698147 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 15.378707ms to process 0 records I0705 14:15:46.698389 1 tasks_processing.go:74] worker 26 stopped. I0705 14:15:46.702065 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-szp5n with fingerprint=3915f28937d310a72dcaac71ec24cb2d88f2a88f73941db0bbc9720800243bc3 I0705 14:15:46.702241 1 recorder.go:75] Recording config/running_containers with fingerprint=94aa232bee589ae17824ce359ac222cd103367cc9f43c5100c2494fcd39373de I0705 14:15:46.702296 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 35.155617ms to process 2 records I0705 14:15:46.702320 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 9.984987ms to process 0 records I0705 14:15:46.702344 1 tasks_processing.go:74] worker 20 stopped. I0705 14:15:46.702430 1 tasks_processing.go:74] worker 24 stopped. I0705 14:15:46.702515 1 recorder.go:75] Recording config/featuregate with fingerprint=50fef8efa5e385dbeadae72b7b81db1841acfbf08a39f91483edf9990bc3cdc6 I0705 14:15:46.702527 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 29.456388ms to process 1 records I0705 14:15:46.702956 1 tasks_processing.go:74] worker 7 stopped. I0705 14:15:46.703033 1 recorder.go:75] Recording config/image with fingerprint=5c8a1435e830a07ae2cea697607c98be2540d49fabb26999bfc1922626c9f45b I0705 14:15:46.703046 1 gather.go:177] gatherer "clusterconfig" function "image" took 37.312417ms to process 1 records I0705 14:15:46.703262 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0705 14:15:46.703341 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0705 14:15:46.703379 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer W0705 14:15:46.703361 1 operator.go:288] started I0705 14:15:46.709098 1 tasks_processing.go:74] worker 2 stopped. I0705 14:15:46.709114 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 11.039161ms to process 0 records I0705 14:15:46.709120 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 11.264513ms to process 0 records I0705 14:15:46.709125 1 tasks_processing.go:74] worker 13 stopped. I0705 14:15:46.709335 1 tasks_processing.go:74] worker 16 stopped. I0705 14:15:46.709499 1 recorder.go:75] Recording config/authentication with fingerprint=1f0f8b34a5d93419e7863fa933dcb604fdf4200189041c2a297434b2dce34366 I0705 14:15:46.709516 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 14.468191ms to process 1 records I0705 14:15:46.712445 1 tasks_processing.go:74] worker 25 stopped. I0705 14:15:46.712596 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=6c5f36c13232562e735e6bd7ab33876d7f16c5abade32673c25d478f6fa11649 I0705 14:15:46.712612 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 22.501925ms to process 1 records I0705 14:15:46.715642 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0705 14:15:46.715653 1 tasks_processing.go:74] worker 10 stopped. I0705 14:15:46.715655 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0705 14:15:46.715664 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0705 14:15:46.715668 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0705 14:15:46.715673 1 controller.go:212] Source scaController *sca.Controller is not ready I0705 14:15:46.715691 1 controller.go:489] The operator is still being initialized I0705 14:15:46.715699 1 controller.go:512] The operator is healthy I0705 14:15:46.715761 1 recorder.go:75] Recording config/ingress with fingerprint=85b8d1f40a1cb38e53c4a5bd932668138692a8b6707eedd9e7ff8e1cee2fdf8f I0705 14:15:46.715774 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 38.612066ms to process 1 records I0705 14:15:46.723268 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0705 14:15:46.723314 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0705 14:15:46.723288 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0705 14:15:46.723689 1 tasks_processing.go:74] worker 6 stopped. I0705 14:15:46.724110 1 recorder.go:75] Recording config/oauth with fingerprint=c193f07821e555f21eb5a9dfa96a494329fdd751a15477274f9386b0e6ea3aee I0705 14:15:46.724155 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 28.712634ms to process 1 records I0705 14:15:46.724228 1 recorder.go:75] Recording config/olm_operators with fingerprint=a93e4a7e4a38a9b2591c0329bebfbc50a9207ae7558af358a864b4167da4b328 I0705 14:15:46.724241 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 60.963694ms to process 1 records I0705 14:15:46.724248 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 60.872311ms to process 0 records E0705 14:15:46.724255 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0705 14:15:46.724261 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 34.873074ms to process 0 records I0705 14:15:46.724261 1 tasks_processing.go:74] worker 21 stopped. I0705 14:15:46.724272 1 tasks_processing.go:74] worker 29 stopped. I0705 14:15:46.724272 1 tasks_processing.go:74] worker 31 stopped. I0705 14:15:46.724322 1 tasks_processing.go:74] worker 17 stopped. I0705 14:15:46.724340 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=0f016bfbc394958fd204fe494e30fe605f14e2ece946430afc721d72dc2ccdc5 I0705 14:15:46.724363 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 47.103754ms to process 1 records I0705 14:15:46.724371 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 42.76079ms to process 0 records I0705 14:15:46.724379 1 tasks_processing.go:74] worker 27 stopped. I0705 14:15:46.724942 1 tasks_processing.go:74] worker 3 stopped. I0705 14:15:46.725260 1 recorder.go:75] Recording config/node/ip-10-0-0-41.ec2.internal with fingerprint=782623fe84b9c037fe788d4fb6b316a63089596431f511c9bf06d70480e31267 I0705 14:15:46.725344 1 recorder.go:75] Recording config/node/ip-10-0-1-166.ec2.internal with fingerprint=9e0ead530242ad12a253a3ea498ec4c348883faef1a96a243737f24c5bc1bbf9 I0705 14:15:46.725418 1 recorder.go:75] Recording config/node/ip-10-0-2-83.ec2.internal with fingerprint=a3dd9fcf47d11a196cfc7949fbae68f2fe41f978da700122ea962634f650e5e4 I0705 14:15:46.725431 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 35.612474ms to process 3 records I0705 14:15:46.725517 1 tasks_processing.go:74] worker 15 stopped. I0705 14:15:46.725631 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=63369f29cc1d99d257c3cf32bd0bf8520d7961ed4da84cc360f7f792a1c92442 I0705 14:15:46.725713 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=adff36d1b64b1a9756bafa086fe9b068308a4076b622ad0dc06f41183e53cba6 I0705 14:15:46.725721 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 62.320323ms to process 2 records I0705 14:15:46.725892 1 tasks_processing.go:74] worker 5 stopped. I0705 14:15:46.726150 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=eedd82c846620b544b3ef8dc9fb4f2361799ed4d3096ae676358f105add59b90 I0705 14:15:46.726160 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.140216ms to process 1 records I0705 14:15:46.726274 1 tasks_processing.go:74] worker 22 stopped. I0705 14:15:46.726288 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 49.035264ms to process 0 records I0705 14:15:46.728279 1 base_controller.go:82] Caches are synced for ConfigController I0705 14:15:46.728297 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0705 14:15:46.731652 1 prometheus_rules.go:88] Prometheus rules successfully created I0705 14:15:46.738411 1 tasks_processing.go:74] worker 4 stopped. I0705 14:15:46.738426 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 58.776298ms to process 0 records I0705 14:15:46.738985 1 configmapobserver.go:84] configmaps "insights-config" not found I0705 14:15:46.739036 1 tasks_processing.go:74] worker 14 stopped. I0705 14:15:46.739495 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=168cf3adc151d49963c33bd07c827d6822795a0ff6a02527b7c9c418dbee05d8 I0705 14:15:46.740033 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=bb0699a25faa8466c9189602982593d8701ca05a90de8485f8fdc9b05a63bdd9 I0705 14:15:46.741705 1 gather.go:177] gatherer "clusterconfig" function "crds" took 40.989924ms to process 2 records I0705 14:15:46.745525 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0705 14:15:46.748884 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.15:60273->172.30.0.10:53: read: connection refused E0705 14:15:46.748892 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2777feeb43-3cc6-4f63-94cc-99fdc664680d%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.15:60273->172.30.0.10:53: read: connection refused I0705 14:15:46.748903 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2777feeb43-3cc6-4f63-94cc-99fdc664680d%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.15:60273->172.30.0.10:53: read: connection refused I0705 14:15:46.748902 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.15:60273->172.30.0.10:53: read: connection refused I0705 14:15:46.758369 1 tasks_processing.go:74] worker 12 stopped. E0705 14:15:46.758382 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0705 14:15:46.758388 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0705 14:15:46.758391 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0705 14:15:46.758399 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0705 14:15:46.758427 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0705 14:15:46.758434 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0705 14:15:46.758438 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0705 14:15:46.758442 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0705 14:15:46.758477 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0705 14:15:46.758485 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0705 14:15:46.758491 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 77.344033ms to process 7 records I0705 14:15:46.761401 1 tasks_processing.go:74] worker 11 stopped. E0705 14:15:46.761413 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0705 14:15:46.761419 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2rc17hprlithorsdlc375b9k20kqu52c-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2rc17hprlithorsdlc375b9k20kqu52c-primary-cert-bundle-secret" not found I0705 14:15:46.761458 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=0d9e9bbe5a384a8b6ce189a49c8f5c0e9e4a8e8050efcb3c6e19eb98d74551c4 I0705 14:15:46.761468 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 66.676752ms to process 1 records I0705 14:15:46.774986 1 tasks_processing.go:74] worker 9 stopped. I0705 14:15:46.775243 1 recorder.go:75] Recording config/version with fingerprint=7cb76e7027ee72f87dc3b4adb3833905408860915a67c857bff1fc97494b798d I0705 14:15:46.775254 1 recorder.go:75] Recording config/id with fingerprint=373393d1eabb1cfc5450a5f45ad07afaa8f87180637b2df2eca5877d6e34c151 I0705 14:15:46.775260 1 gather.go:177] gatherer "clusterconfig" function "version" took 111.143729ms to process 2 records I0705 14:15:46.791626 1 tasks_processing.go:74] worker 28 stopped. I0705 14:15:46.791640 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 111.017988ms to process 0 records I0705 14:15:46.803538 1 base_controller.go:82] Caches are synced for LoggingSyncer I0705 14:15:46.803573 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0705 14:15:46.825053 1 tasks_processing.go:74] worker 18 stopped. I0705 14:15:46.825093 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0705 14:15:46.825113 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 155.883783ms to process 1 records I0705 14:15:47.101070 1 gather_cluster_operator_pods_and_events.go:121] Found 16 pods with 18 containers I0705 14:15:47.101084 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1398101 bytes I0705 14:15:47.101303 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-99wvf pod in namespace openshift-dns (previous: false). I0705 14:15:47.332567 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-99wvf pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-99wvf\" is waiting to start: ContainerCreating" I0705 14:15:47.332584 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-99wvf\" is waiting to start: ContainerCreating" I0705 14:15:47.332593 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-99wvf pod in namespace openshift-dns (previous: false). I0705 14:15:47.506959 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-99wvf pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-99wvf\" is waiting to start: ContainerCreating" I0705 14:15:47.506977 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-99wvf\" is waiting to start: ContainerCreating" I0705 14:15:47.507008 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-pm4kf pod in namespace openshift-dns (previous: false). W0705 14:15:47.693681 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0705 14:15:47.728789 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pm4kf pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-pm4kf\" is waiting to start: ContainerCreating" I0705 14:15:47.728805 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-pm4kf\" is waiting to start: ContainerCreating" I0705 14:15:47.728816 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-pm4kf pod in namespace openshift-dns (previous: false). I0705 14:15:47.922924 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-pm4kf pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-pm4kf\" is waiting to start: ContainerCreating" I0705 14:15:47.922942 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-pm4kf\" is waiting to start: ContainerCreating" I0705 14:15:47.922951 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-gdzd4 pod in namespace openshift-dns (previous: false). I0705 14:15:48.105626 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0705 14:15:48.105644 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-hxjbk pod in namespace openshift-dns (previous: false). I0705 14:15:48.139705 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0705 14:15:48.331226 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-hxjbk pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "container \"dns-node-resolver\" in pod \"node-resolver-hxjbk\" is waiting to start: ContainerCreating" I0705 14:15:48.331246 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns-node-resolver\" in pod \"node-resolver-hxjbk\" is waiting to start: ContainerCreating" I0705 14:15:48.331257 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-qtnt4 pod in namespace openshift-dns (previous: false). I0705 14:15:48.513123 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0705 14:15:48.513183 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5864d8cf55-h28qc pod in namespace openshift-image-registry (previous: false). I0705 14:15:48.541044 1 tasks_processing.go:74] worker 8 stopped. I0705 14:15:48.541091 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=b94308cb87accdf018c09444c2dcbb153826f68282b93afc2c3b71a1e0914364 I0705 14:15:48.541118 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=ea748c493739f6717e6d6cf0103c1e4d6cbf665fa42031bdfc6bb0d586607ad8 I0705 14:15:48.541148 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0705 14:15:48.541176 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=1f54b478295e2a0604d8e7cddaf6aed18fdcccb866ad38006e3b79ea5ab4b097 I0705 14:15:48.541209 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0705 14:15:48.541234 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=2e9595b7bed4ad111e4192f72d70e6a85163166874dc4c8683dc4cfaa3d0e0cc I0705 14:15:48.541264 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=6011c51dfffe7b3fa710d2302d04061e1720a8dfc99e7965f0664e9a74e933ca I0705 14:15:48.541288 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=6f2b383778dc2197934f2243ad3df0b30fce6e7b9c561c063cbff273ca4da699 I0705 14:15:48.541304 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=adb346dd181bd0c0a7cb81acb40e245d31f3f2d209311842f2d987c9335f6ea5 I0705 14:15:48.541322 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=deefa4a3f32797cb40c423965ffefa957d6ee156421b0aee5157410901fa9c0b I0705 14:15:48.541332 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0705 14:15:48.541347 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=8ec3114387987b7a327a96e1558faeec1aa01971c4e5f5728309912bc1069c40 I0705 14:15:48.541359 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0705 14:15:48.541374 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=43fafd45ebac4db7ffc3c970a89538cfd3a9e6d1421e025e14284fd4352fa817 I0705 14:15:48.541384 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0705 14:15:48.541397 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=384ba1b4d9a06dfc6c306a1c11d83bfb6450097ad048297c473d7b94ae00aa9a I0705 14:15:48.541406 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0705 14:15:48.541421 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=85a311266b1624d5ed967d67bed0a5948476d20f575746d877f783eb508ff4ea I0705 14:15:48.541566 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=9a1bce1ce2517036163e08a0ef20984b5f50dabfac90784598f9f70d607bcaec I0705 14:15:48.541578 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0705 14:15:48.541586 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0705 14:15:48.541607 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0705 14:15:48.541629 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=edd45fd6c2aed0382812ee8c3a0c90fd83efcb2a382c46b3f8b18b89eac6e3b4 I0705 14:15:48.541653 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=a7fb28e4ba083f0d5e9db6b381132b954c8bcd8b6f4daafc4d718267634e8d83 I0705 14:15:48.541662 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0705 14:15:48.541678 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=86608c6f3c575b01d0b054a106ef0bb29b08d23f7fffba5bc6af15a8b6151c65 I0705 14:15:48.541687 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0705 14:15:48.541700 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=49af7390cbc1a57f5ea8a4274959eee799ac4abaaea4e2004893311f59402bd9 I0705 14:15:48.541714 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=26c9b6580bd4f9ec8791a7690588fef0ebd2b575dc5ba29cc656898feda9b828 I0705 14:15:48.541730 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=a19bb7f40b7ec717a80feb1d95f0991b9d7a9b1858574580fbee47e96d2d7628 I0705 14:15:48.541745 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=adad5f538ed33945af13856934d1957148650a3b8b4ad672a9f0363cc73a62fc I0705 14:15:48.541777 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=b13b1e07d991d67ac59fdd0cdbb35ea944d7691e5d2723a3a229ab182ca40b3c I0705 14:15:48.541802 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=9d5ae74f50168e264cd92349367f27df7c4c0ef208ec4b65e81cd2d8ff31f224 I0705 14:15:48.541818 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0705 14:15:48.541827 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0705 14:15:48.541833 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.876250476s to process 35 records W0705 14:15:48.692738 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0705 14:15:48.707271 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-5864d8cf55-h28qc pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-5864d8cf55-h28qc\" is waiting to start: ContainerCreating" I0705 14:15:48.707286 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-5864d8cf55-h28qc\" is waiting to start: ContainerCreating" I0705 14:15:48.707326 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-59f5569d96-ngxxm pod in namespace openshift-image-registry (previous: false). I0705 14:15:48.907579 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-59f5569d96-ngxxm pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-59f5569d96-ngxxm\" is waiting to start: ContainerCreating" I0705 14:15:48.907597 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-59f5569d96-ngxxm\" is waiting to start: ContainerCreating" I0705 14:15:48.907654 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-59f5569d96-z7qhv pod in namespace openshift-image-registry (previous: false). I0705 14:15:49.101029 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0705 14:15:49.101045 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-dw72l pod in namespace openshift-image-registry (previous: false). I0705 14:15:49.307329 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0705 14:15:49.307347 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-hkgth pod in namespace openshift-image-registry (previous: false). I0705 14:15:49.510525 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-ca-hkgth pod in namespace openshift-image-registry for failing operator node-ca (previous: false): "container \"node-ca\" in pod \"node-ca-hkgth\" is waiting to start: ContainerCreating" I0705 14:15:49.510540 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"node-ca\" in pod \"node-ca-hkgth\" is waiting to start: ContainerCreating" I0705 14:15:49.510570 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-qx86d pod in namespace openshift-image-registry (previous: false). W0705 14:15:49.693167 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0705 14:15:49.705533 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0705 14:15:49.705574 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-597fcb5bbc-p4djh pod in namespace openshift-ingress (previous: false). I0705 14:15:49.905646 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-597fcb5bbc-p4djh pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-597fcb5bbc-p4djh\" is waiting to start: ContainerCreating" I0705 14:15:49.905671 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-597fcb5bbc-p4djh\" is waiting to start: ContainerCreating" I0705 14:15:49.905685 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6f5c46f44b-4zn48 pod in namespace openshift-ingress (previous: false). I0705 14:15:50.108336 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6f5c46f44b-4zn48 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6f5c46f44b-4zn48\" is waiting to start: ContainerCreating" I0705 14:15:50.108353 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6f5c46f44b-4zn48\" is waiting to start: ContainerCreating" I0705 14:15:50.108363 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6f5c46f44b-7xzwj pod in namespace openshift-ingress (previous: false). I0705 14:15:50.312236 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6f5c46f44b-7xzwj pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6f5c46f44b-7xzwj\" is waiting to start: ContainerCreating" I0705 14:15:50.312253 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6f5c46f44b-7xzwj\" is waiting to start: ContainerCreating" I0705 14:15:50.312292 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-p2cch pod in namespace openshift-ingress-canary (previous: false). I0705 14:15:50.510464 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-p2cch pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-p2cch\" is waiting to start: ContainerCreating" I0705 14:15:50.510481 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-p2cch\" is waiting to start: ContainerCreating" I0705 14:15:50.510509 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-zshjt pod in namespace openshift-ingress-canary (previous: false). W0705 14:15:50.692351 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0705 14:15:50.707213 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-zshjt pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-zshjt\" is waiting to start: ContainerCreating" I0705 14:15:50.707228 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-zshjt\" is waiting to start: ContainerCreating" I0705 14:15:50.707250 1 tasks_processing.go:74] worker 1 stopped. I0705 14:15:50.707351 1 recorder.go:75] Recording events/openshift-dns with fingerprint=f3ee34f11c15947debde92969de9025393f3c7f73d5e7eff22dc7171b820bfa9 I0705 14:15:50.707456 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=a3563eb01f4ae1607a53252ff43906ab3b6c071d62b7fb6ba2922655af6a5c89 I0705 14:15:50.707494 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=c1bb12272f07ef66bbce8b278310b128ded744700e42ac0f06a03bcfbee6d9fc I0705 14:15:50.707571 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=7e28abfa0df459ad1734605612f57a9ead06daa4f23afaf40667b1cc2ec5ff1e I0705 14:15:50.707594 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=3843e3967df48c176d1c85ae9d7e1805a68800dd6263118bdab1b32a02bf1213 I0705 14:15:50.707735 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-pm4kf with fingerprint=40a2656208f62aed185400ea18d6a9b5793e664ba866cab6967e8078d1ca2acf I0705 14:15:50.707859 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5864d8cf55-h28qc with fingerprint=e485045e5933bc50dd6c583624c8fb33e7b96fc19b01ad6164a8ebabbef73495 I0705 14:15:50.707962 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-59f5569d96-ngxxm with fingerprint=91662a68dd66b404af7f47e2b2e26eafa6e771e9afa80fc37ba08f6e88bd276c I0705 14:15:50.708047 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-59f5569d96-z7qhv with fingerprint=ad5ba65171599ffbf5b195c49f7b473706d347e3833688a89bdad5ebef42d6ac I0705 14:15:50.708116 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-p2cch with fingerprint=8ab345596c0e728edca034c73788557a739a9e34bcfa6fbc1edb23fc470d7905 I0705 14:15:50.708181 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-zshjt with fingerprint=bb753433b05bfb18743946815e86ab7c9cd0a899f59d6c82049c37811200a931 I0705 14:15:50.708194 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.044126249s to process 11 records W0705 14:15:51.692385 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0705 14:15:51.692411 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0705 14:15:51.692430 1 tasks_processing.go:74] worker 19 stopped. E0705 14:15:51.692442 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0705 14:15:51.692455 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0705 14:15:51.692475 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0705 14:15:51.692491 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.028706884s to process 1 records I0705 14:15:59.295071 1 tasks_processing.go:74] worker 23 stopped. I0705 14:15:59.295119 1 recorder.go:75] Recording config/installplans with fingerprint=e3c18441424aaa81f54ea9723d194eacb5824eb47178047bd893de11825bc8a2 I0705 14:15:59.295133 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.63152876s to process 1 records I0705 14:15:59.846000 1 configmapobserver.go:84] configmaps "insights-config" not found I0705 14:16:00.083786 1 tasks_processing.go:74] worker 0 stopped. I0705 14:16:00.084123 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=a7a4de02df1cc1abdf9107bf21a9d837b0f58083251a7f44b02bf3cc6249ecc3 I0705 14:16:00.084146 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.407169764s to process 1 records E0705 14:16:00.084226 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.422s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0705 14:16:00.085360 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0705 14:16:00.085379 1 periodic.go:209] Running workloads gatherer I0705 14:16:00.085398 1 tasks_processing.go:45] number of workers: 2 I0705 14:16:00.085416 1 tasks_processing.go:69] worker 1 listening for tasks. I0705 14:16:00.085422 1 tasks_processing.go:71] worker 1 working on workload_info task. I0705 14:16:00.085433 1 tasks_processing.go:69] worker 0 listening for tasks. I0705 14:16:00.085451 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0705 14:16:00.111433 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0705 14:16:00.116094 1 tasks_processing.go:74] worker 0 stopped. I0705 14:16:00.116114 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 30.617321ms to process 0 records I0705 14:16:00.120350 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (10ms) I0705 14:16:00.129227 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0705 14:16:00.137896 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0705 14:16:00.146530 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0705 14:16:00.155487 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (9ms) I0705 14:16:00.164521 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (9ms) I0705 14:16:00.172767 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (8ms) I0705 14:16:00.183444 1 gather_workloads_info.go:387] No image sha256:88437b35173fe2e8a4d9f148ec17c754bee4df3f723b3a5676955eef4f363d31 (11ms) I0705 14:16:00.192094 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (9ms) I0705 14:16:00.200743 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (9ms) I0705 14:16:00.219590 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (19ms) I0705 14:16:00.327456 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (108ms) I0705 14:16:00.425584 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (98ms) I0705 14:16:00.521758 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (96ms) I0705 14:16:00.622499 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (101ms) I0705 14:16:00.720287 1 gather_workloads_info.go:387] No image sha256:d3209a996e5ce362136e70a0bcf06d89973905de73e234b644d8ae8886d067f1 (98ms) I0705 14:16:00.824084 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (104ms) I0705 14:16:00.917424 1 configmapobserver.go:84] configmaps "insights-config" not found I0705 14:16:00.920589 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (96ms) I0705 14:16:01.020453 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0705 14:16:01.117330 1 configmapobserver.go:84] configmaps "insights-config" not found I0705 14:16:01.120483 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0705 14:16:01.221948 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0705 14:16:01.330920 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (109ms) I0705 14:16:01.422047 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (91ms) I0705 14:16:01.522147 1 gather_workloads_info.go:387] No image sha256:ef2bfb546cd8aac31b7671585f718f44b268ed43a9c8562016ab741558a7dfa1 (100ms) I0705 14:16:01.620680 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (99ms) I0705 14:16:01.722662 1 gather_workloads_info.go:387] No image sha256:fab570ecdc9c3620b032bc4c5b44240f5ebac0bb9e48dfeea32bcd36eb462e5a (102ms) I0705 14:16:01.826703 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (104ms) I0705 14:16:01.921020 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (94ms) I0705 14:16:02.022852 1 gather_workloads_info.go:387] No image sha256:a505dd5b62d744c2cfda2666b5e21e0a4704354a3114488bd3291456085e1e33 (102ms) I0705 14:16:02.125679 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (103ms) I0705 14:16:02.222309 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (97ms) I0705 14:16:02.320714 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (98ms) I0705 14:16:02.419836 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (99ms) I0705 14:16:02.521618 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (102ms) I0705 14:16:02.621908 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0705 14:16:02.726692 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (105ms) I0705 14:16:02.820473 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (94ms) I0705 14:16:02.920456 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0705 14:16:03.020949 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0705 14:16:03.020981 1 tasks_processing.go:74] worker 1 stopped. E0705 14:16:03.020993 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0705 14:16:03.021309 1 recorder.go:75] Recording config/workload_info with fingerprint=a9b5057857d88a6179506f428fda19a7437af6c37474a59e00ac510a17479114 I0705 14:16:03.021326 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.935549427s to process 1 records E0705 14:16:03.021348 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.935s with: function \"workload_info\" failed with an error" I0705 14:16:03.022449 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0705 14:16:03.022460 1 periodic.go:209] Running conditional gatherer I0705 14:16:03.029113 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0705 14:16:03.035713 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.15:46330->172.30.0.10:53: read: connection refused E0705 14:16:03.035961 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0705 14:16:03.036019 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0705 14:16:03.042283 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0705 14:16:03.042296 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042301 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042305 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042308 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042311 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042314 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042317 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042319 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0705 14:16:03.042322 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0705 14:16:03.042338 1 tasks_processing.go:45] number of workers: 3 I0705 14:16:03.042350 1 tasks_processing.go:69] worker 2 listening for tasks. I0705 14:16:03.042356 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0705 14:16:03.042359 1 tasks_processing.go:69] worker 0 listening for tasks. I0705 14:16:03.042369 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0705 14:16:03.042375 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0705 14:16:03.042377 1 tasks_processing.go:69] worker 1 listening for tasks. I0705 14:16:03.042385 1 tasks_processing.go:74] worker 1 stopped. I0705 14:16:03.042463 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0705 14:16:03.042492 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 904ns to process 1 records I0705 14:16:03.042542 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0705 14:16:03.042567 1 tasks_processing.go:74] worker 0 stopped. I0705 14:16:03.042574 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.043µs to process 1 records I0705 14:16:03.042694 1 tasks_processing.go:74] worker 2 stopped. I0705 14:16:03.042706 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 312.369µs to process 0 records I0705 14:16:03.042722 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.15:46330->172.30.0.10:53: read: connection refused I0705 14:16:03.042738 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0705 14:16:03.067821 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=6ef57a47811c06be124d3e310cd996ea352ad0c332ddec7c639d07d90d9e3a88 I0705 14:16:03.067938 1 diskrecorder.go:70] Writing 106 records to /var/lib/insights-operator/insights-2026-07-05-141603.tar.gz I0705 14:16:03.074723 1 diskrecorder.go:51] Wrote 106 records to disk in 6ms I0705 14:16:03.074751 1 periodic.go:278] Gathering cluster info every 2h0m0s I0705 14:16:03.074764 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0705 14:16:13.254280 1 configmapobserver.go:84] configmaps "insights-config" not found I0705 14:17:01.340759 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="c85cb935a2fa1449ca75260cce9dcdd5edcb642418cb9038276def39ec999ea4") W0705 14:17:01.340795 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0705 14:17:01.340848 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="7a313ec1c407596d077faeaf77ef588e3017ff65ee805bf46388b7e394703096") I0705 14:17:01.340845 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0705 14:17:01.340885 1 periodic.go:170] Shutting down I0705 14:17:01.340908 1 base_controller.go:181] Shutting down LoggingSyncer ... I0705 14:17:01.340945 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0705 14:17:01.340971 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0705 14:17:01.340987 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0705 14:17:01.340980 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0705 14:17:01.340973 1 base_controller.go:181] Shutting down ConfigController ... E0705 14:17:01.341042 1 controller.go:299] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0705 14:17:01.341049 1 base_controller.go:113] All ConfigController workers have been terminated I0705 14:17:01.341108 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0705 14:17:01.341120 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0705 14:17:01.340982 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0705 14:17:01.340950 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="4aca34b9a14f2d9064aa21c9e2641b7bca6c97946fa765969981acdd0c28c30c") I0705 14:17:01.341024 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0705 14:17:01.341025 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain"