I0416 19:18:43.669095 1 cmd.go:253] Using service-serving-cert provided certificates I0416 19:18:43.669603 1 observer_polling.go:159] Starting file observer I0416 19:18:44.751591 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0416 19:18:44.751783 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0416 19:18:44.752327 1 secure_serving.go:57] Forcing use of http/1.1 only W0416 19:18:44.752346 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0416 19:18:44.752351 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0416 19:18:44.752356 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0416 19:18:44.752358 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0416 19:18:44.752361 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0416 19:18:44.752364 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0416 19:18:44.752375 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0416 19:18:44.756861 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0416 19:18:44.756936 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"d4e61c0a-4b3b-44a0-a55a-53ff6ee97bcb", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0416 19:18:44.757416 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0416 19:18:44.757435 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0416 19:18:44.757433 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0416 19:18:44.757448 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0416 19:18:44.757460 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0416 19:18:44.757467 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0416 19:18:44.757716 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0416 19:18:44.757989 1 secure_serving.go:213] Serving securely on [::]:8443 I0416 19:18:44.758016 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0416 19:18:44.767416 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0416 19:18:44.767442 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0416 19:18:44.767544 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0416 19:18:44.772890 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:18:44.772919 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0416 19:18:44.777868 1 secretconfigobserver.go:119] support secret does not exist I0416 19:18:44.782873 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:18:44.787367 1 secretconfigobserver.go:119] support secret does not exist I0416 19:18:44.790963 1 recorder.go:161] Pruning old reports every 5h38m35s, max age is 288h0m0s I0416 19:18:44.796568 1 operator.go:316] The last pod state is unhealthy I0416 19:18:44.857639 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0416 19:18:44.857657 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0416 19:18:44.857741 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0416 19:18:44.867954 1 base_controller.go:82] Caches are synced for ConfigController I0416 19:18:44.867969 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0416 19:18:44.873322 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:19:04.800405 1 operator.go:316] The last pod state is unhealthy I0416 19:19:24.799891 1 operator.go:316] The last pod state is unhealthy I0416 19:19:44.797870 1 operator.go:316] The last pod state is unhealthy I0416 19:20:04.798896 1 operator.go:316] The last pod state is unhealthy I0416 19:20:24.801152 1 operator.go:316] The last pod state is unhealthy I0416 19:20:44.799748 1 operator.go:316] The last pod state is unhealthy I0416 19:21:04.807904 1 operator.go:316] The last pod state is unhealthy I0416 19:21:24.801740 1 operator.go:316] The last pod state is unhealthy I0416 19:21:44.800214 1 operator.go:316] The last pod state is unhealthy I0416 19:22:04.799121 1 operator.go:316] The last pod state is unhealthy I0416 19:22:24.800091 1 operator.go:316] The last pod state is unhealthy I0416 19:22:44.798244 1 operator.go:316] The last pod state is unhealthy I0416 19:23:04.804484 1 operator.go:316] The last pod state is unhealthy I0416 19:23:24.801247 1 operator.go:316] The last pod state is unhealthy I0416 19:23:44.795720 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:23:44.796707 1 operator.go:316] The last pod state is unhealthy I0416 19:23:44.800271 1 secretconfigobserver.go:119] support secret does not exist I0416 19:23:57.898116 1 operator.go:248] Unable to check insights-operator pod status. Setting initial delay to 13m1.947146821s I0416 19:23:57.898245 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0416 19:23:57.898272 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 14m31s I0416 19:23:57.898248 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0416 19:23:57.898313 1 insightsreport.go:296] Starting report retriever I0416 19:23:57.898318 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0416 19:23:57.908341 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0416 19:23:57.908360 1 controller.go:326] The initial operator extension status is healthy I0416 19:23:57.908368 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:23:57.908372 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:23:57.908375 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:23:57.908391 1 controller.go:489] The operator is still being initialized I0416 19:23:57.908396 1 controller.go:512] The operator is healthy I0416 19:23:57.908479 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0416 19:23:57.908487 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0416 19:23:57.908627 1 operator.go:288] started I0416 19:23:57.908649 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0416 19:23:57.914425 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:23:57.914436 1 controller.go:212] Source scaController *sca.Controller is not ready I0416 19:23:57.914440 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0416 19:23:57.914443 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:23:57.914445 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:23:57.914460 1 controller.go:489] The operator is still being initialized I0416 19:23:57.914465 1 controller.go:512] The operator is healthy I0416 19:23:57.924314 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0416 19:23:58.009193 1 base_controller.go:82] Caches are synced for LoggingSyncer I0416 19:23:58.009226 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0416 19:23:58.011966 1 cluster_transfer.go:102] no available accepted cluster transfer I0416 19:23:58.011986 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0416 19:23:58.235597 1 sca.go:266] etc-pki-entitlement secret successfully created I0416 19:23:58.235618 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0416 19:25:57.915728 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:25:57.915748 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:25:57.915752 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:25:57.915772 1 controller.go:489] The operator is still being initialized I0416 19:25:57.915777 1 controller.go:512] The operator is healthy I0416 19:27:57.919534 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:27:57.919550 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:27:57.919553 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:27:57.919574 1 controller.go:512] The operator is healthy I0416 19:28:44.784068 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:28:44.788942 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:28:44.801029 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0416 19:28:44.806121 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:28:44.806140 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0416 19:28:44.810622 1 secretconfigobserver.go:119] support secret does not exist I0416 19:29:57.918842 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:29:57.918860 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:29:57.918867 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:29:57.918918 1 controller.go:512] The operator is healthy I0416 19:31:57.918870 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:31:57.918889 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:31:57.918892 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:31:57.918916 1 controller.go:512] The operator is healthy