W0326 23:18:06.419057 1 cmd.go:257] Using insecure, self-signed certificates I0326 23:18:06.899039 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:06.899369 1 observer_polling.go:159] Starting file observer I0326 23:18:07.426440 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0326 23:18:07.426746 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0326 23:18:07.427223 1 secure_serving.go:57] Forcing use of http/1.1 only W0326 23:18:07.427247 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0326 23:18:07.427252 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0326 23:18:07.427257 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0326 23:18:07.427260 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0326 23:18:07.427264 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0326 23:18:07.427269 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0326 23:18:07.427618 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0326 23:18:07.431225 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0326 23:18:07.431265 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"967252c9-9a4f-4bcf-b637-828c45f44e2a", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0326 23:18:07.431979 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0326 23:18:07.432000 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0326 23:18:07.431993 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0326 23:18:07.432003 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0326 23:18:07.432014 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0326 23:18:07.432014 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0326 23:18:07.432311 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-2651654095/tls.crt::/tmp/serving-cert-2651654095/tls.key" I0326 23:18:07.432547 1 secure_serving.go:213] Serving securely on [::]:8443 I0326 23:18:07.432581 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0326 23:18:07.435669 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0326 23:18:07.435695 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0326 23:18:07.435800 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0326 23:18:07.440354 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0326 23:18:07.440371 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0326 23:18:07.448021 1 secretconfigobserver.go:119] support secret does not exist I0326 23:18:07.451981 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0326 23:18:07.455860 1 secretconfigobserver.go:119] support secret does not exist I0326 23:18:07.459125 1 recorder.go:161] Pruning old reports every 7h30m37s, max age is 288h0m0s I0326 23:18:07.463841 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0326 23:18:07.463861 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0326 23:18:07.463864 1 periodic.go:209] Running clusterconfig gatherer I0326 23:18:07.463868 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0326 23:18:07.463881 1 insightsreport.go:296] Starting report retriever I0326 23:18:07.463889 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0326 23:18:07.463901 1 tasks_processing.go:45] number of workers: 64 I0326 23:18:07.463929 1 tasks_processing.go:69] worker 3 listening for tasks. I0326 23:18:07.463942 1 tasks_processing.go:69] worker 0 listening for tasks. I0326 23:18:07.463944 1 tasks_processing.go:69] worker 10 listening for tasks. I0326 23:18:07.463950 1 tasks_processing.go:69] worker 1 listening for tasks. I0326 23:18:07.463957 1 tasks_processing.go:69] worker 4 listening for tasks. I0326 23:18:07.463960 1 tasks_processing.go:69] worker 2 listening for tasks. I0326 23:18:07.463960 1 tasks_processing.go:69] worker 21 listening for tasks. I0326 23:18:07.463968 1 tasks_processing.go:69] worker 12 listening for tasks. I0326 23:18:07.463970 1 tasks_processing.go:69] worker 22 listening for tasks. I0326 23:18:07.463972 1 tasks_processing.go:71] worker 21 working on machine_autoscalers task. I0326 23:18:07.463977 1 tasks_processing.go:69] worker 13 listening for tasks. I0326 23:18:07.463978 1 tasks_processing.go:69] worker 23 listening for tasks. I0326 23:18:07.463979 1 tasks_processing.go:69] worker 5 listening for tasks. I0326 23:18:07.463973 1 tasks_processing.go:69] worker 15 listening for tasks. I0326 23:18:07.463986 1 tasks_processing.go:69] worker 14 listening for tasks. I0326 23:18:07.463990 1 tasks_processing.go:69] worker 47 listening for tasks. I0326 23:18:07.463964 1 tasks_processing.go:71] worker 4 working on infrastructures task. I0326 23:18:07.463994 1 tasks_processing.go:69] worker 6 listening for tasks. I0326 23:18:07.463985 1 tasks_processing.go:69] worker 45 listening for tasks. I0326 23:18:07.463993 1 tasks_processing.go:69] worker 35 listening for tasks. I0326 23:18:07.464000 1 tasks_processing.go:69] worker 11 listening for tasks. I0326 23:18:07.464008 1 tasks_processing.go:69] worker 37 listening for tasks. I0326 23:18:07.463997 1 tasks_processing.go:69] worker 48 listening for tasks. I0326 23:18:07.464001 1 tasks_processing.go:69] worker 36 listening for tasks. I0326 23:18:07.464017 1 tasks_processing.go:69] worker 38 listening for tasks. I0326 23:18:07.464020 1 tasks_processing.go:69] worker 8 listening for tasks. I0326 23:18:07.464018 1 tasks_processing.go:69] worker 58 listening for tasks. I0326 23:18:07.464026 1 tasks_processing.go:71] worker 0 working on openstack_controlplanes task. I0326 23:18:07.464031 1 tasks_processing.go:69] worker 19 listening for tasks. I0326 23:18:07.464034 1 tasks_processing.go:69] worker 40 listening for tasks. I0326 23:18:07.464031 1 tasks_processing.go:69] worker 63 listening for tasks. I0326 23:18:07.464037 1 tasks_processing.go:69] worker 50 listening for tasks. I0326 23:18:07.464011 1 tasks_processing.go:69] worker 60 listening for tasks. I0326 23:18:07.464011 1 tasks_processing.go:69] worker 7 listening for tasks. I0326 23:18:07.464044 1 tasks_processing.go:69] worker 51 listening for tasks. I0326 23:18:07.464009 1 tasks_processing.go:69] worker 34 listening for tasks. I0326 23:18:07.464043 1 tasks_processing.go:69] worker 9 listening for tasks. I0326 23:18:07.464055 1 tasks_processing.go:69] worker 52 listening for tasks. I0326 23:18:07.464056 1 tasks_processing.go:69] worker 18 listening for tasks. I0326 23:18:07.464057 1 tasks_processing.go:69] worker 43 listening for tasks. I0326 23:18:07.464059 1 tasks_processing.go:69] worker 44 listening for tasks. I0326 23:18:07.464056 1 tasks_processing.go:69] worker 31 listening for tasks. I0326 23:18:07.464066 1 tasks_processing.go:69] worker 20 listening for tasks. I0326 23:18:07.464002 1 tasks_processing.go:69] worker 49 listening for tasks. I0326 23:18:07.464069 1 tasks_processing.go:69] worker 55 listening for tasks. I0326 23:18:07.464075 1 tasks_processing.go:69] worker 27 listening for tasks. I0326 23:18:07.464076 1 tasks_processing.go:71] worker 20 working on validating_webhook_configurations task. I0326 23:18:07.464021 1 tasks_processing.go:69] worker 62 listening for tasks. I0326 23:18:07.464097 1 tasks_processing.go:69] worker 54 listening for tasks. I0326 23:18:07.464100 1 tasks_processing.go:71] worker 55 working on image task. I0326 23:18:07.464102 1 tasks_processing.go:71] worker 62 working on oauths task. I0326 23:18:07.464081 1 tasks_processing.go:71] worker 10 working on pdbs task. I0326 23:18:07.464104 1 tasks_processing.go:69] worker 28 listening for tasks. I0326 23:18:07.464111 1 tasks_processing.go:71] worker 47 working on storage_cluster task. I0326 23:18:07.464024 1 tasks_processing.go:69] worker 24 listening for tasks. I0326 23:18:07.464040 1 tasks_processing.go:69] worker 16 listening for tasks. I0326 23:18:07.464043 1 tasks_processing.go:69] worker 41 listening for tasks. I0326 23:18:07.464046 1 tasks_processing.go:69] worker 30 listening for tasks. I0326 23:18:07.464047 1 tasks_processing.go:69] worker 17 listening for tasks. I0326 23:18:07.464050 1 tasks_processing.go:69] worker 42 listening for tasks. I0326 23:18:07.464053 1 tasks_processing.go:69] worker 29 listening for tasks. I0326 23:18:07.464016 1 tasks_processing.go:69] worker 61 listening for tasks. I0326 23:18:07.463963 1 tasks_processing.go:71] worker 1 working on sap_pods task. I0326 23:18:07.464138 1 tasks_processing.go:71] worker 48 working on support_secret task. I0326 23:18:07.463970 1 tasks_processing.go:71] worker 2 working on ingress_certificates task. I0326 23:18:07.464256 1 tasks_processing.go:71] worker 23 working on feature_gates task. I0326 23:18:07.464299 1 tasks_processing.go:71] worker 60 working on image_pruners task. I0326 23:18:07.464308 1 tasks_processing.go:71] worker 14 working on cost_management_metrics_configs task. I0326 23:18:07.464313 1 tasks_processing.go:71] worker 12 working on machine_healthchecks task. I0326 23:18:07.464319 1 tasks_processing.go:71] worker 13 working on clusterroles task. I0326 23:18:07.464028 1 tasks_processing.go:69] worker 39 listening for tasks. I0326 23:18:07.464067 1 tasks_processing.go:69] worker 32 listening for tasks. I0326 23:18:07.464070 1 tasks_processing.go:69] worker 53 listening for tasks. I0326 23:18:07.464074 1 tasks_processing.go:69] worker 26 listening for tasks. I0326 23:18:07.464076 1 tasks_processing.go:71] worker 49 working on nodenetworkstates task. I0326 23:18:07.464377 1 tasks_processing.go:71] worker 36 working on node_logs task. I0326 23:18:07.464432 1 tasks_processing.go:71] worker 43 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0326 23:18:07.464076 1 tasks_processing.go:69] worker 56 listening for tasks. I0326 23:18:07.464534 1 tasks_processing.go:71] worker 24 working on jaegers task. I0326 23:18:07.464543 1 tasks_processing.go:71] worker 42 working on machine_configs task. I0326 23:18:07.464270 1 tasks_processing.go:71] worker 15 working on networks task. I0326 23:18:07.464006 1 tasks_processing.go:69] worker 59 listening for tasks. I0326 23:18:07.464688 1 tasks_processing.go:71] worker 59 working on crds task. I0326 23:18:07.464545 1 tasks_processing.go:71] worker 39 working on machines task. I0326 23:18:07.464023 1 tasks_processing.go:71] worker 3 working on olm_operators task. I0326 23:18:07.464533 1 tasks_processing.go:71] worker 26 working on machine_sets task. I0326 23:18:07.464075 1 tasks_processing.go:69] worker 33 listening for tasks. I0326 23:18:07.464940 1 tasks_processing.go:71] worker 33 working on machine_config_pools task. I0326 23:18:07.464080 1 tasks_processing.go:71] worker 27 working on authentication task. I0326 23:18:07.464103 1 tasks_processing.go:69] worker 57 listening for tasks. I0326 23:18:07.465071 1 tasks_processing.go:71] worker 57 working on openstack_dataplanenodesets task. I0326 23:18:07.464107 1 tasks_processing.go:71] worker 38 working on tsdb_status task. I0326 23:18:07.464122 1 tasks_processing.go:71] worker 45 working on nodes task. W0326 23:18:07.465126 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:07.465147 1 tasks_processing.go:71] worker 38 working on overlapping_namespace_uids task. I0326 23:18:07.464114 1 tasks_processing.go:71] worker 6 working on schedulers task. I0326 23:18:07.465180 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 38.819µs to process 0 records I0326 23:18:07.464126 1 tasks_processing.go:71] worker 35 working on nodenetworkconfigurationpolicies task. I0326 23:18:07.464131 1 tasks_processing.go:71] worker 11 working on aggregated_monitoring_cr_names task. I0326 23:18:07.464135 1 tasks_processing.go:71] worker 37 working on ceph_cluster task. I0326 23:18:07.463985 1 tasks_processing.go:69] worker 46 listening for tasks. I0326 23:18:07.464279 1 tasks_processing.go:71] worker 5 working on openstack_version task. I0326 23:18:07.465517 1 tasks_processing.go:71] worker 46 working on container_images task. I0326 23:18:07.464281 1 tasks_processing.go:71] worker 7 working on openshift_logging task. I0326 23:18:07.464286 1 tasks_processing.go:71] worker 52 working on sap_config task. I0326 23:18:07.464290 1 tasks_processing.go:71] worker 63 working on storage_classes task. I0326 23:18:07.464292 1 tasks_processing.go:71] worker 40 working on ingress task. I0326 23:18:07.464290 1 tasks_processing.go:71] worker 51 working on version task. I0326 23:18:07.464294 1 tasks_processing.go:71] worker 50 working on dvo_metrics task. I0326 23:18:07.464297 1 tasks_processing.go:71] worker 34 working on pod_network_connectivity_checks task. I0326 23:18:07.464297 1 tasks_processing.go:71] worker 58 working on image_registries task. I0326 23:18:07.464301 1 tasks_processing.go:71] worker 9 working on mutating_webhook_configurations task. I0326 23:18:07.464302 1 tasks_processing.go:71] worker 19 working on cluster_apiserver task. I0326 23:18:07.464309 1 tasks_processing.go:71] worker 22 working on metrics task. W0326 23:18:07.466516 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:07.464450 1 tasks_processing.go:71] worker 44 working on service_accounts task. I0326 23:18:07.464456 1 tasks_processing.go:71] worker 18 working on proxies task. I0326 23:18:07.464463 1 tasks_processing.go:71] worker 31 working on openstack_dataplanedeployments task. I0326 23:18:07.464523 1 tasks_processing.go:71] worker 17 working on lokistack task. I0326 23:18:07.464527 1 tasks_processing.go:71] worker 54 working on install_plans task. I0326 23:18:07.464531 1 tasks_processing.go:71] worker 28 working on container_runtime_configs task. I0326 23:18:07.464539 1 tasks_processing.go:71] worker 56 working on operators_pods_and_events task. I0326 23:18:07.464550 1 tasks_processing.go:71] worker 29 working on openshift_machine_api_events task. I0326 23:18:07.464554 1 tasks_processing.go:71] worker 61 working on silenced_alerts task. I0326 23:18:07.464563 1 tasks_processing.go:71] worker 41 working on qemu_kubevirt_launcher_logs task. I0326 23:18:07.464567 1 tasks_processing.go:71] worker 16 working on sap_datahubs task. I0326 23:18:07.464064 1 tasks_processing.go:69] worker 25 listening for tasks. I0326 23:18:07.464572 1 tasks_processing.go:71] worker 32 working on certificate_signing_requests task. I0326 23:18:07.464578 1 tasks_processing.go:71] worker 30 working on operators task. I0326 23:18:07.464578 1 tasks_processing.go:71] worker 53 working on config_maps task. I0326 23:18:07.464286 1 tasks_processing.go:71] worker 8 working on active_alerts task. W0326 23:18:07.466588 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:07.466597 1 tasks_processing.go:71] worker 22 working on monitoring_persistent_volumes task. I0326 23:18:07.466818 1 tasks_processing.go:74] worker 25 stopped. I0326 23:18:07.466610 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 33.359µs to process 0 records I0326 23:18:07.467102 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 68.684µs to process 0 records I0326 23:18:07.467115 1 tasks_processing.go:74] worker 8 stopped. W0326 23:18:07.467148 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:07.467165 1 tasks_processing.go:74] worker 61 stopped. I0326 23:18:07.467177 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 38.679µs to process 0 records I0326 23:18:07.467553 1 tasks_processing.go:74] worker 21 stopped. I0326 23:18:07.467570 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 3.567602ms to process 0 records I0326 23:18:07.468281 1 tasks_processing.go:74] worker 1 stopped. I0326 23:18:07.468294 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 4.132528ms to process 0 records I0326 23:18:07.468839 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0326 23:18:07.468860 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0326 23:18:07.468867 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0326 23:18:07.468871 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0326 23:18:07.468889 1 controller.go:489] The operator is still being initialized I0326 23:18:07.468900 1 controller.go:512] The operator is healthy I0326 23:18:07.470606 1 tasks_processing.go:74] worker 49 stopped. I0326 23:18:07.470620 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 6.253071ms to process 0 records I0326 23:18:07.470626 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 6.574891ms to process 0 records I0326 23:18:07.470631 1 tasks_processing.go:74] worker 0 stopped. I0326 23:18:07.470742 1 tasks_processing.go:74] worker 12 stopped. E0326 23:18:07.470754 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0326 23:18:07.470762 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 6.417218ms to process 0 records I0326 23:18:07.475809 1 tasks_processing.go:74] worker 47 stopped. I0326 23:18:07.475826 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 11.688621ms to process 0 records I0326 23:18:07.475859 1 tasks_processing.go:74] worker 24 stopped. I0326 23:18:07.475870 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 11.307645ms to process 0 records I0326 23:18:07.487572 1 tasks_processing.go:74] worker 35 stopped. I0326 23:18:07.487588 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 22.323678ms to process 0 records E0326 23:18:07.487598 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0326 23:18:07.487607 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 23.435743ms to process 0 records I0326 23:18:07.487616 1 tasks_processing.go:74] worker 48 stopped. I0326 23:18:07.487690 1 tasks_processing.go:74] worker 5 stopped. I0326 23:18:07.487704 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 22.181737ms to process 0 records I0326 23:18:07.487712 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 22.236106ms to process 0 records I0326 23:18:07.487718 1 tasks_processing.go:74] worker 37 stopped. I0326 23:18:07.487726 1 tasks_processing.go:74] worker 7 stopped. I0326 23:18:07.487737 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 22.148661ms to process 0 records I0326 23:18:07.487826 1 tasks_processing.go:74] worker 26 stopped. I0326 23:18:07.487844 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 22.942424ms to process 0 records I0326 23:18:07.487852 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 22.742882ms to process 0 records I0326 23:18:07.487857 1 tasks_processing.go:74] worker 57 stopped. I0326 23:18:07.488074 1 tasks_processing.go:74] worker 17 stopped. I0326 23:18:07.488099 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 21.143542ms to process 0 records I0326 23:18:07.488167 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.765368ms to process 0 records I0326 23:18:07.488180 1 tasks_processing.go:74] worker 14 stopped. I0326 23:18:07.488203 1 tasks_processing.go:74] worker 55 stopped. I0326 23:18:07.488826 1 recorder.go:75] Recording config/image with fingerprint=acb8dc2906cdba6a25e02d9bd249cd14233fd2810c474c2c95efc0dd0493701d I0326 23:18:07.488845 1 gather.go:177] gatherer "clusterconfig" function "image" took 24.090841ms to process 1 records I0326 23:18:07.488853 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 22.616107ms to process 0 records I0326 23:18:07.488863 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 21.902576ms to process 0 records I0326 23:18:07.488872 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 21.509872ms to process 0 records I0326 23:18:07.488878 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 24.311729ms to process 0 records I0326 23:18:07.488881 1 tasks_processing.go:74] worker 52 stopped. I0326 23:18:07.488896 1 tasks_processing.go:74] worker 29 stopped. I0326 23:18:07.488896 1 tasks_processing.go:74] worker 22 stopped. I0326 23:18:07.488913 1 tasks_processing.go:74] worker 36 stopped. I0326 23:18:07.488996 1 tasks_processing.go:74] worker 60 stopped. I0326 23:18:07.490236 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c908213ca19d01527e6c8671f02edf8e1ff10a3bd5b10129d5207157c6e8a442 I0326 23:18:07.490262 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 24.489362ms to process 1 records I0326 23:18:07.490504 1 tasks_processing.go:74] worker 4 stopped. I0326 23:18:07.491881 1 recorder.go:75] Recording config/infrastructure with fingerprint=8dd7a2b7a15a834fd76a1eb5107b6c1ba1b97a8c75c346ecee8ddf99a54b761c I0326 23:18:07.491904 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 25.500006ms to process 1 records I0326 23:18:07.492081 1 recorder.go:75] Recording config/ingress with fingerprint=6441263ed73c5960b664ff2449b71cb17cfd00d5f1f9697a879ac1b1293e6ba3 I0326 23:18:07.492136 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 24.255679ms to process 1 records I0326 23:18:07.492148 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 25.438083ms to process 0 records I0326 23:18:07.492116 1 tasks_processing.go:74] worker 40 stopped. I0326 23:18:07.492182 1 tasks_processing.go:74] worker 3 stopped. I0326 23:18:07.492264 1 tasks_processing.go:74] worker 15 stopped. I0326 23:18:07.492344 1 recorder.go:75] Recording config/network with fingerprint=6f9338a51fb636461d10ca2b2c2660af8bb5db13653797b92b57cc30484d901b I0326 23:18:07.492361 1 gather.go:177] gatherer "clusterconfig" function "networks" took 27.188842ms to process 1 records E0326 23:18:07.492371 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0326 23:18:07.492384 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.174489ms to process 0 records E0326 23:18:07.492393 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0326 23:18:07.492399 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 25.897807ms to process 0 records I0326 23:18:07.492429 1 tasks_processing.go:74] worker 39 stopped. I0326 23:18:07.492454 1 tasks_processing.go:74] worker 34 stopped. I0326 23:18:07.492472 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=6a0c1a410d36cdd22ce0202d2aa845a4468a850578285a5424f64eda1a364406 I0326 23:18:07.492483 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 26.9089ms to process 1 records I0326 23:18:07.492496 1 tasks_processing.go:74] worker 6 stopped. I0326 23:18:07.492590 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=e3ceba325710278d8dc4ef41e7bdd201d30b02e550b6bcdbb57b2e02029e2807 I0326 23:18:07.492604 1 tasks_processing.go:74] worker 63 stopped. I0326 23:18:07.492618 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=d063db524fa1dd00a3efa1eaaa644cabbca2a7ae1bb3b4f842fabeb9fc7b7c28 I0326 23:18:07.492625 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 26.727285ms to process 2 records I0326 23:18:07.492733 1 tasks_processing.go:74] worker 9 stopped. I0326 23:18:07.492834 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=44fcd8a70154382859b6b9606337508f2ace59e30ef6dfb467154f48751a7af6 I0326 23:18:07.492911 1 gather_logs.go:145] no pods in namespace were found I0326 23:18:07.492915 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=b7826c1e0145dc27e49dd9c9a6df79a640d0c58b717fc6c68e5025d3d23e78e9 I0326 23:18:07.493001 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=f7acc04e282ddc9d84e07db088dab5c9aca5f24bf01769cb4464723ca7494eb6 I0326 23:18:07.493017 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 26.105335ms to process 3 records I0326 23:18:07.493114 1 tasks_processing.go:74] worker 23 stopped. I0326 23:18:07.493203 1 recorder.go:75] Recording config/featuregate with fingerprint=e225a530c2e3ae1b1dbc487ccd1f8f15acf53e50a330eca676ad9162b098be38 I0326 23:18:07.493220 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 28.215803ms to process 1 records I0326 23:18:07.493309 1 tasks_processing.go:74] worker 27 stopped. I0326 23:18:07.493427 1 recorder.go:75] Recording config/authentication with fingerprint=cf858df716460998fb068f6e91db91de9242b485385dbb56fc94e180bd311e64 I0326 23:18:07.493442 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 27.636274ms to process 1 records I0326 23:18:07.493515 1 tasks_processing.go:74] worker 19 stopped. I0326 23:18:07.493596 1 recorder.go:75] Recording config/apiserver with fingerprint=3b6b4926accb1a4a793ecee2c269e9b86eaaff9e4fa908ddc1c53ffb0bfd43de I0326 23:18:07.493608 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 26.255747ms to process 1 records I0326 23:18:07.493614 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 26.252246ms to process 0 records I0326 23:18:07.493618 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 26.223333ms to process 0 records I0326 23:18:07.493642 1 tasks_processing.go:74] worker 16 stopped. I0326 23:18:07.493656 1 tasks_processing.go:74] worker 31 stopped. I0326 23:18:07.493785 1 tasks_processing.go:74] worker 58 stopped. I0326 23:18:07.494015 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=c8916f39c415be8ac908772f770f8f55205ccb7e50de4de3f723f04e098946db I0326 23:18:07.494037 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 26.8508ms to process 1 records I0326 23:18:07.494046 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 25.793544ms to process 0 records I0326 23:18:07.494098 1 tasks_processing.go:74] worker 41 stopped. I0326 23:18:07.494162 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=fa2564e6b976576c09f072aa96032cb0a40ec83ab760b399516341a60190e1bb I0326 23:18:07.494174 1 tasks_processing.go:74] worker 10 stopped. I0326 23:18:07.495153 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=7db9ee4b984546f2ddb4abe5d4e49bbe9e206c9073f82de07fd9e8f88c306c0b I0326 23:18:07.495234 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=c26159c3ff1f4907bf2c104f3a1c869633e2bcc660678dc971060fd508281bf1 I0326 23:18:07.495307 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 28.825117ms to process 3 records I0326 23:18:07.495467 1 recorder.go:75] Recording config/proxy with fingerprint=d35f836da29e8dd1010d404678e757b4724ad39a4b9880631ad5e1cd4b0871d0 I0326 23:18:07.495531 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 26.227837ms to process 1 records I0326 23:18:07.495486 1 tasks_processing.go:74] worker 18 stopped. I0326 23:18:07.495662 1 tasks_processing.go:74] worker 62 stopped. I0326 23:18:07.496143 1 recorder.go:75] Recording config/oauth with fingerprint=4fd192ecb0e4749c6a186d9ed84d249ee9a22ae8a728175796b3b72206e23f6b I0326 23:18:07.496201 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 29.174934ms to process 1 records I0326 23:18:07.496231 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 26.872819ms to process 0 records I0326 23:18:07.496257 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 28.54265ms to process 0 records I0326 23:18:07.496325 1 tasks_processing.go:74] worker 28 stopped. I0326 23:18:07.496329 1 tasks_processing.go:74] worker 33 stopped. I0326 23:18:07.496545 1 tasks_processing.go:74] worker 45 stopped. I0326 23:18:07.497353 1 recorder.go:75] Recording config/node/ip-10-0-0-147.ec2.internal with fingerprint=b3b8304ffd6309c4bb8435f816705c1b63bb6872919cae8a31eecf94d82a965a I0326 23:18:07.497564 1 recorder.go:75] Recording config/node/ip-10-0-1-110.ec2.internal with fingerprint=809802da8a9a8abca802ac933845908c83cf8d609dadfd2f2314e0c7d2daf2a1 I0326 23:18:07.497764 1 recorder.go:75] Recording config/node/ip-10-0-2-206.ec2.internal with fingerprint=4328ef330393f8c114126ac988fc4e3b96ffc5a2eaf21e4eee3f5728b4b7815c I0326 23:18:07.497790 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 30.602211ms to process 3 records I0326 23:18:07.498706 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0326 23:18:07.498727 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0326 23:18:07.498815 1 tasks_processing.go:74] worker 38 stopped. W0326 23:18:07.498826 1 operator.go:288] started I0326 23:18:07.498836 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0326 23:18:07.498840 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0326 23:18:07.498926 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 33.653617ms to process 1 records I0326 23:18:07.499057 1 tasks_processing.go:74] worker 20 stopped. I0326 23:18:07.499131 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=40bfd7c81c1306b572ff4c50b30df5252990cd906d154b5c9b4599310c39efc8 I0326 23:18:07.499241 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=0f1ba97183774e9fa8c52a88f9619106f918466ce7696f2041d5c39794a57205 I0326 23:18:07.499265 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=c84dab48505a6912e983b649978fd831159615884247e913cc54c9592d4d224c I0326 23:18:07.499288 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=90e738c63c41551604bee45fd54860d8d4d95f10fe06db1a4e643deac1808a6c I0326 23:18:07.499312 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=b6372109d72501b8e198edc1b007d6bec822fa254a4b63e2ec6479cd6881b2a2 I0326 23:18:07.499348 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=2c43da024e06c8693ab8e431aafdca23808ce64dc1f20d0216aad0874036e18c I0326 23:18:07.499384 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=85ac4f8a24935697947144b0bfafe3ec168bc13ef5be83ad6bfbe05c8665ff25 I0326 23:18:07.499441 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=9a7741d331e7c2b9132effbe28ccfa2b3995b07facec0db82e53a37d80018d25 I0326 23:18:07.499480 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=f9927daf7b905034602fd47f35c5f87aa147bbab87661e043b0a0906dafe13d6 I0326 23:18:07.499540 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=1498aba02ceea58b99307347e72ce7181ebf1f003cc561329afaf8dbe5f26740 I0326 23:18:07.499581 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=d061e9154f3b025bcdb6bd057a0c3d7201abff531a25b8b96045f43aceb38f4e I0326 23:18:07.499618 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 34.826472ms to process 11 records W0326 23:18:07.501321 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0326 23:18:07.501373 1 tasks_processing.go:74] worker 32 stopped. I0326 23:18:07.501394 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 34.74044ms to process 0 records I0326 23:18:07.506583 1 tasks_processing.go:74] worker 13 stopped. I0326 23:18:07.506791 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=474ec861d26534a88d14905905baa7068d5dd53d78901d889fd8ac52303c8160 I0326 23:18:07.506909 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=f0a3cfb288d0bcb1d00d74e8e0e5660810276d1f0ceb8c7d211a69551f59083b I0326 23:18:07.506926 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 42.247132ms to process 2 records I0326 23:18:07.506940 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 41.410333ms to process 0 records I0326 23:18:07.506949 1 tasks_processing.go:74] worker 11 stopped. I0326 23:18:07.507481 1 controller.go:212] Source scaController *sca.Controller is not ready I0326 23:18:07.507532 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0326 23:18:07.507540 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0326 23:18:07.507543 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0326 23:18:07.507545 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0326 23:18:07.507562 1 controller.go:489] The operator is still being initialized I0326 23:18:07.507570 1 controller.go:512] The operator is healthy I0326 23:18:07.507882 1 tasks_processing.go:74] worker 59 stopped. I0326 23:18:07.508674 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=52f76900f48f77a15c66ea38c7a725b11e0b871bd0ab7609c04180fc1f249b78 I0326 23:18:07.508930 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=03bafdba15a26a837e8a75efd371f61727572dcf1c284e05218d5da1afdfa3c9 I0326 23:18:07.508945 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.182359ms to process 2 records I0326 23:18:07.513161 1 tasks_processing.go:74] worker 46 stopped. I0326 23:18:07.513426 1 recorder.go:75] Recording config/running_containers with fingerprint=428d5e61503aac3ac8f015a616f42071ce4025a843e355c96d514f6d0d037198 I0326 23:18:07.513489 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 47.63097ms to process 1 records I0326 23:18:07.513520 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 48.761138ms to process 0 records I0326 23:18:07.513552 1 tasks_processing.go:74] worker 43 stopped. I0326 23:18:07.518719 1 prometheus_rules.go:88] Prometheus rules successfully created E0326 23:18:07.518792 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%278cc947eb-bc1a-4072-b489-5cdb42ef9972%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:42478->172.30.0.10:53: read: connection refused I0326 23:18:07.518845 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%278cc947eb-bc1a-4072-b489-5cdb42ef9972%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:42478->172.30.0.10:53: read: connection refused I0326 23:18:07.532280 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0326 23:18:07.532298 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0326 23:18:07.532314 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0326 23:18:07.536288 1 base_controller.go:82] Caches are synced for ConfigController I0326 23:18:07.536303 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0326 23:18:07.552731 1 configmapobserver.go:84] configmaps "insights-config" not found I0326 23:18:07.558001 1 tasks_processing.go:74] worker 53 stopped. E0326 23:18:07.558021 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0326 23:18:07.558029 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0326 23:18:07.558033 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0326 23:18:07.558045 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0326 23:18:07.558081 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0326 23:18:07.558112 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0326 23:18:07.558118 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0326 23:18:07.558123 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0326 23:18:07.558182 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0326 23:18:07.558194 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0326 23:18:07.558204 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 91.156565ms to process 7 records I0326 23:18:07.572389 1 tasks_processing.go:74] worker 51 stopped. I0326 23:18:07.572823 1 recorder.go:75] Recording config/version with fingerprint=4bd0babc7154cf4e27d28c73cd8316ce4d45e674ad4a4d47ec470c3dad74c4ad I0326 23:18:07.572875 1 recorder.go:75] Recording config/id with fingerprint=d53594825d61908589891e82aaca4b39e35e2fb22b039ddb26dbf108b1a6023b I0326 23:18:07.572891 1 gather.go:177] gatherer "clusterconfig" function "version" took 106.601174ms to process 2 records I0326 23:18:07.582395 1 tasks_processing.go:74] worker 42 stopped. I0326 23:18:07.582439 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0326 23:18:07.582460 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 117.76031ms to process 1 records I0326 23:18:07.588820 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0326 23:18:07.592708 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:54285->172.30.0.10:53: read: connection refused I0326 23:18:07.592725 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:54285->172.30.0.10:53: read: connection refused I0326 23:18:07.599684 1 base_controller.go:82] Caches are synced for LoggingSyncer I0326 23:18:07.599702 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0326 23:18:07.628549 1 tasks_processing.go:74] worker 2 stopped. E0326 23:18:07.628574 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0326 23:18:07.628586 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p9mp45l21c084apimipppb2mg5li0kr-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p9mp45l21c084apimipppb2mg5li0kr-primary-cert-bundle-secret" not found I0326 23:18:07.628648 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=02d51386b9f4d6657935d75f55f9eee329803359bf630e31777ac54c15fb8bdf I0326 23:18:07.628662 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 164.281543ms to process 1 records I0326 23:18:07.906960 1 gather_cluster_operator_pods_and_events.go:121] Found 16 pods with 18 containers I0326 23:18:07.906973 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1398101 bytes I0326 23:18:07.907312 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-7bvdm pod in namespace openshift-dns (previous: false). I0326 23:18:08.138231 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-7bvdm pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-7bvdm\" is waiting to start: ContainerCreating" I0326 23:18:08.138249 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-7bvdm\" is waiting to start: ContainerCreating" I0326 23:18:08.138259 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-7bvdm pod in namespace openshift-dns (previous: false). I0326 23:18:08.311995 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-7bvdm pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-7bvdm\" is waiting to start: ContainerCreating" I0326 23:18:08.312020 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-7bvdm\" is waiting to start: ContainerCreating" I0326 23:18:08.312033 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-s9x84 pod in namespace openshift-dns (previous: false). W0326 23:18:08.496778 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0326 23:18:08.530248 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-s9x84 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-s9x84\" is waiting to start: ContainerCreating" I0326 23:18:08.530266 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-s9x84\" is waiting to start: ContainerCreating" I0326 23:18:08.530276 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-s9x84 pod in namespace openshift-dns (previous: false). I0326 23:18:08.713868 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-s9x84 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-s9x84\" is waiting to start: ContainerCreating" I0326 23:18:08.713884 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-s9x84\" is waiting to start: ContainerCreating" I0326 23:18:08.713894 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-9gvrd pod in namespace openshift-dns (previous: false). I0326 23:18:08.913826 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0326 23:18:08.913843 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-s5gnk pod in namespace openshift-dns (previous: false). I0326 23:18:08.924760 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0326 23:18:09.112129 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0326 23:18:09.112146 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-wd9rz pod in namespace openshift-dns (previous: false). I0326 23:18:09.327697 1 tasks_processing.go:74] worker 30 stopped. I0326 23:18:09.327748 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=030016e865dfb3d3648d91f71fb522db1c8a1d28583cccbae77ee3f15a7c554e I0326 23:18:09.327779 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=7b2625f2cd94f5a547bd6f2d8e68c50cbe9ac042f230477a74f4fbfdc277f17b I0326 23:18:09.327808 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0326 23:18:09.327832 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=3616dff277470c0a4a7d7a4f5700f7bad31c3779f5d886486ef884e26622e30d I0326 23:18:09.327848 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0326 23:18:09.327872 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=13cfe21735d2ee114878089f59a9587c23ef1accdba8296fa9601c1d5c090da1 I0326 23:18:09.327919 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=64315713a671ffa43cd0fe8bbde39c03e154937a8539024d223529aea6dd8a0f I0326 23:18:09.327943 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=7ae97c17d97d231ce6291bf565acdfd1a40e14df9cf0c6a9cfaa0d3eb862388b I0326 23:18:09.327957 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=ae86d9e7e28d31c20f1fcd852b8dc73f0d94be1d7d798cb06b9c729fe4785a68 I0326 23:18:09.327974 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=bd4dbd0684c5b887e027a855f5ec970e2ab3961d51545cb7752c1faf54c564f8 I0326 23:18:09.327982 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0326 23:18:09.327998 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=79e71550fada0ed220fcb52e532881fb0002a367f42b638f3641151c7864e756 I0326 23:18:09.328008 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0326 23:18:09.328024 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=8babb5302fbba2eb73379d7dc0c27fea038476a552079065b80b1e58582c521d I0326 23:18:09.328033 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0326 23:18:09.328045 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=b0489ff2d3dcf6644a870373d9a94d3b796106d1627a742028cab354ae386b4a I0326 23:18:09.328054 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0326 23:18:09.328069 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=c9e0a36b18ed27e32c65cecb42c289301ef7c28fb9e3d9baaf8184cf7e55b1ac I0326 23:18:09.328232 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=7a854101b1fa727f20f827773c3cbb2a7f2f5c0da59c04dd061458448b5ce2b2 I0326 23:18:09.328242 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0326 23:18:09.328249 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0326 23:18:09.328269 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0326 23:18:09.328291 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=69e08de37502356193d7a789fab66a15fb2b146d52995d0d57ab682fe7825e8e I0326 23:18:09.328315 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=44a83cd535f7187a615e45cd6c732902cc18b5cbbbbfa1b2a1903564d00a2268 I0326 23:18:09.328325 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0326 23:18:09.328341 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=28674fa78d1844e9661ac46d1f1325fee31d800c97b158671217638f5b414505 I0326 23:18:09.328350 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0326 23:18:09.328367 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=5b28aa3f42b8db4a3b7d02af3a48eb51e01e8efedb1d832516c83290c2c310d3 I0326 23:18:09.328382 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=3640430669f7584ead457be0358c52454e44388d24480f241a79b1c5b3a58540 I0326 23:18:09.328398 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=f207f57368b71a924d64b51d53aab1cbb8989cd5acddf23f94fc250bcddd4c28 I0326 23:18:09.328414 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=f0b02e60f902a17f7985f330853ef679a6a46932d3cfc5f4e5072740139b04fc I0326 23:18:09.328427 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=c7b059db631ba5148d6010efb321078b73514caefe10ca83287a700abe6e4897 I0326 23:18:09.328451 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=96fa1ebb1c697a82c2060d109a0003c2d4f87393a8cfd73244585477d960ebe8 I0326 23:18:09.328466 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0326 23:18:09.328474 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0326 23:18:09.328481 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.860953221s to process 35 records I0326 23:18:09.352167 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-wd9rz pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "container \"dns-node-resolver\" in pod \"node-resolver-wd9rz\" is waiting to start: ContainerCreating" I0326 23:18:09.352184 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns-node-resolver\" in pod \"node-resolver-wd9rz\" is waiting to start: ContainerCreating" I0326 23:18:09.352195 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6b6976d69f-6j4zl pod in namespace openshift-image-registry (previous: false). W0326 23:18:09.496617 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0326 23:18:09.511312 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6b6976d69f-6j4zl pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6b6976d69f-6j4zl\" is waiting to start: ContainerCreating" I0326 23:18:09.511328 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6b6976d69f-6j4zl\" is waiting to start: ContainerCreating" I0326 23:18:09.511339 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6b6976d69f-z6wcc pod in namespace openshift-image-registry (previous: false). I0326 23:18:09.712846 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6b6976d69f-z6wcc pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6b6976d69f-z6wcc\" is waiting to start: ContainerCreating" I0326 23:18:09.712863 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6b6976d69f-z6wcc\" is waiting to start: ContainerCreating" I0326 23:18:09.712873 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6b6985c588-cdlbt pod in namespace openshift-image-registry (previous: false). I0326 23:18:09.911966 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0326 23:18:09.911984 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-qw8z8 pod in namespace openshift-image-registry (previous: false). I0326 23:18:10.112890 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0326 23:18:10.112907 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-r5hx4 pod in namespace openshift-image-registry (previous: false). I0326 23:18:10.313202 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-ca-r5hx4 pod in namespace openshift-image-registry for failing operator node-ca (previous: false): "container \"node-ca\" in pod \"node-ca-r5hx4\" is waiting to start: ContainerCreating" I0326 23:18:10.313219 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"node-ca\" in pod \"node-ca-r5hx4\" is waiting to start: ContainerCreating" I0326 23:18:10.313229 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-xtclc pod in namespace openshift-image-registry (previous: false). W0326 23:18:10.495986 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0326 23:18:10.516249 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0326 23:18:10.516264 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5684c7485-njxfd pod in namespace openshift-ingress (previous: false). I0326 23:18:10.712393 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5684c7485-njxfd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5684c7485-njxfd\" is waiting to start: ContainerCreating" I0326 23:18:10.712411 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5684c7485-njxfd\" is waiting to start: ContainerCreating" I0326 23:18:10.712422 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8cf49676f-cqxgh pod in namespace openshift-ingress (previous: false). I0326 23:18:10.911620 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-8cf49676f-cqxgh pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-8cf49676f-cqxgh\" is waiting to start: ContainerCreating" I0326 23:18:10.911641 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-8cf49676f-cqxgh\" is waiting to start: ContainerCreating" I0326 23:18:10.911653 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-8cf49676f-pprxk pod in namespace openshift-ingress (previous: false). I0326 23:18:11.112418 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-8cf49676f-pprxk pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-8cf49676f-pprxk\" is waiting to start: ContainerCreating" I0326 23:18:11.112437 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-8cf49676f-pprxk\" is waiting to start: ContainerCreating" I0326 23:18:11.112447 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-gt7gr pod in namespace openshift-ingress-canary (previous: false). I0326 23:18:11.311870 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-gt7gr pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-gt7gr\" is waiting to start: ContainerCreating" I0326 23:18:11.311889 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-gt7gr\" is waiting to start: ContainerCreating" I0326 23:18:11.311905 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-zd6kj pod in namespace openshift-ingress-canary (previous: false). W0326 23:18:11.496877 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0326 23:18:11.516499 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-zd6kj pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-zd6kj\" is waiting to start: ContainerCreating" I0326 23:18:11.516521 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-zd6kj\" is waiting to start: ContainerCreating" I0326 23:18:11.516537 1 tasks_processing.go:74] worker 56 stopped. I0326 23:18:11.516655 1 recorder.go:75] Recording events/openshift-dns with fingerprint=b9509374b67c8b1b9d705615b548919beaa21a20d1e0d452121924b57b5e8a51 I0326 23:18:11.516771 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=8ba2e47d4007eb57641fa4517271e3d70f039995ce28f8039580c87460325e9d I0326 23:18:11.516798 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=5f893920e438ed91af73e06d58ae4baa1604e62e591a93cfaea76397c9e1d3b0 I0326 23:18:11.516843 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=cd3623f27b0e30c7a49391eeb98387f401bc75214091bd72c7e8851076ff3b1a I0326 23:18:11.516856 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=3e1edbbeaed0fcad01fe0b3bad7241a145b69ee5553e2f66baa2b3c2f9afe284 I0326 23:18:11.516864 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.049738803s to process 5 records W0326 23:18:12.492982 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0326 23:18:12.493007 1 tasks_processing.go:74] worker 50 stopped. E0326 23:18:12.493019 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0326 23:18:12.493029 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0326 23:18:12.493043 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0326 23:18:12.493053 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.027154919s to process 1 records I0326 23:18:19.904966 1 tasks_processing.go:74] worker 54 stopped. I0326 23:18:19.905003 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0326 23:18:19.905014 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.437830859s to process 1 records I0326 23:18:20.612797 1 configmapobserver.go:84] configmaps "insights-config" not found I0326 23:18:20.673246 1 tasks_processing.go:74] worker 44 stopped. I0326 23:18:20.673498 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=5f3315562ab922847b80910efb7b38b1b30885aa3cdda0895bd6e02e66f43fde I0326 23:18:20.673515 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.206698056s to process 1 records E0326 23:18:20.673587 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.209s with: function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0326 23:18:20.674700 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0326 23:18:20.674716 1 periodic.go:209] Running workloads gatherer I0326 23:18:20.674730 1 tasks_processing.go:45] number of workers: 2 I0326 23:18:20.674738 1 tasks_processing.go:69] worker 1 listening for tasks. I0326 23:18:20.674742 1 tasks_processing.go:71] worker 1 working on workload_info task. I0326 23:18:20.674749 1 tasks_processing.go:69] worker 0 listening for tasks. I0326 23:18:20.674765 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0326 23:18:20.697173 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0326 23:18:20.702447 1 tasks_processing.go:74] worker 0 stopped. I0326 23:18:20.702463 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 27.66579ms to process 0 records I0326 23:18:20.705684 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0326 23:18:20.715658 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (10ms) I0326 23:18:20.724923 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0326 23:18:20.735414 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (10ms) I0326 23:18:20.743903 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (8ms) I0326 23:18:20.751656 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (8ms) I0326 23:18:20.759434 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (8ms) I0326 23:18:20.774508 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (15ms) I0326 23:18:20.785233 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (11ms) I0326 23:18:20.793232 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (8ms) I0326 23:18:20.805632 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (12ms) I0326 23:18:20.908642 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (103ms) I0326 23:18:21.007140 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (98ms) I0326 23:18:21.106631 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (99ms) I0326 23:18:21.208309 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (102ms) I0326 23:18:21.307178 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (99ms) I0326 23:18:21.414687 1 gather_workloads_info.go:387] No image sha256:d3209a996e5ce362136e70a0bcf06d89973905de73e234b644d8ae8886d067f1 (107ms) I0326 23:18:21.507077 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (92ms) I0326 23:18:21.606883 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0326 23:18:21.706424 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0326 23:18:21.806076 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0326 23:18:21.906151 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0326 23:18:22.006321 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0326 23:18:22.106991 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0326 23:18:22.206248 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0326 23:18:22.306493 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (100ms) I0326 23:18:22.335870 1 configmapobserver.go:84] configmaps "insights-config" not found I0326 23:18:22.408502 1 gather_workloads_info.go:387] No image sha256:ef2bfb546cd8aac31b7671585f718f44b268ed43a9c8562016ab741558a7dfa1 (102ms) I0326 23:18:22.506910 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (98ms) I0326 23:18:22.539110 1 configmapobserver.go:84] configmaps "insights-config" not found I0326 23:18:22.606507 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0326 23:18:22.706234 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0326 23:18:22.810031 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (104ms) I0326 23:18:22.909624 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0326 23:18:23.006756 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (97ms) I0326 23:18:23.106336 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0326 23:18:23.205939 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0326 23:18:23.205971 1 tasks_processing.go:74] worker 1 stopped. E0326 23:18:23.205981 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0326 23:18:23.206241 1 recorder.go:75] Recording config/workload_info with fingerprint=13708b0a9c8b54cf5bca6029d2b56cf7a6aeb988923e6b407bfbd2395c8a02c1 I0326 23:18:23.206256 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.531222401s to process 1 records E0326 23:18:23.206280 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.531s with: function \"workload_info\" failed with an error" I0326 23:18:23.207383 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0326 23:18:23.207395 1 periodic.go:209] Running conditional gatherer I0326 23:18:23.213055 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0326 23:18:23.219724 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:38692->172.30.0.10:53: read: connection refused E0326 23:18:23.219969 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0326 23:18:23.220044 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0326 23:18:23.225969 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0326 23:18:23.225982 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.225987 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.225990 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.225993 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.225996 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.225999 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.226002 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.226005 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0326 23:18:23.226008 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0326 23:18:23.226024 1 tasks_processing.go:45] number of workers: 3 I0326 23:18:23.226036 1 tasks_processing.go:69] worker 2 listening for tasks. I0326 23:18:23.226042 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0326 23:18:23.226044 1 tasks_processing.go:69] worker 0 listening for tasks. I0326 23:18:23.226054 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0326 23:18:23.226057 1 tasks_processing.go:69] worker 1 listening for tasks. I0326 23:18:23.226066 1 tasks_processing.go:74] worker 1 stopped. I0326 23:18:23.226060 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0326 23:18:23.226138 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0326 23:18:23.226153 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 750ns to process 1 records I0326 23:18:23.226188 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0326 23:18:23.226197 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.11µs to process 1 records I0326 23:18:23.226203 1 tasks_processing.go:74] worker 0 stopped. I0326 23:18:23.226516 1 tasks_processing.go:74] worker 2 stopped. I0326 23:18:23.226558 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 440.542µs to process 0 records I0326 23:18:23.226640 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:38692->172.30.0.10:53: read: connection refused I0326 23:18:23.226668 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0326 23:18:23.248257 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=b554398f04206c4293c7f5b5be980962b3f5451a4bcc50837ce2b38d4afe8e20 I0326 23:18:23.248380 1 diskrecorder.go:70] Writing 98 records to /var/lib/insights-operator/insights-2026-03-26-231823.tar.gz I0326 23:18:23.254062 1 diskrecorder.go:51] Wrote 98 records to disk in 5ms I0326 23:18:23.254104 1 periodic.go:278] Gathering cluster info every 2h0m0s I0326 23:18:23.254121 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0326 23:18:37.098402 1 configmapobserver.go:84] configmaps "insights-config" not found I0326 23:19:31.899785 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="01d5af9b142860cfffe117e22ed0eaa00ee291d695aedda564def139898a6054") W0326 23:19:31.899819 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0326 23:19:31.899865 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="c87c956d6472ee7e20b2e52dfd7d7eb122611c3ee3d05650bb83448243028976") I0326 23:19:31.899893 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0326 23:19:31.899930 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0326 23:19:31.899930 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0326 23:19:31.899948 1 base_controller.go:181] Shutting down ConfigController ... I0326 23:19:31.899954 1 base_controller.go:181] Shutting down LoggingSyncer ... I0326 23:19:31.899963 1 periodic.go:170] Shutting down I0326 23:19:31.899959 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="f9f7e4625e0afc214b7ef00127ea77539af00eb6a7540cef5ff780b896807539") I0326 23:19:31.899975 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0326 23:19:31.899979 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0326 23:19:31.899982 1 base_controller.go:113] All ConfigController workers have been terminated I0326 23:19:31.899990 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0326 23:19:31.899997 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0326 23:19:31.900001 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0326 23:19:31.900018 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0326 23:19:31.900032 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting"