INFO: Log in to your Red Hat account... INFO: Configure AWS Credentials... INFO: Logged in as 'konflux-ci-418295695583' on 'https://api.openshift.com' INFO: Create ROSA with HCP cluster... WARN: The version of Red Hat OpenShift Service on AWS that you are installing will no longer be supported after '2025-11-02'. Red Hat recommends selecting a newer version. For more information, see https://docs.openshift.com/rosa/rosa_policy/rosa-life-cycle.html time=2025-09-04T05:36:45Z level=info msg=Ignored check for policy key 'sts_hcp_ec2_registry_permission_policy' (zero egress feature toggle is not enabled) INFO: Creating cluster 'kx-6c470fb64a' INFO: To view a list of clusters and their status, run 'rosa list clusters' INFO: Cluster 'kx-6c470fb64a' has been created. INFO: Once the cluster is installed you will need to add an Identity Provider before you can login into the cluster. See 'rosa create idp --help' for more information. Name: kx-6c470fb64a Domain Prefix: kx-6c470fb64a Display Name: kx-6c470fb64a ID: 2l3desots79l2etc1eu2s7g9hup91a91 External ID: 5feefd51-5292-4795-a9d9-92a132ca77ad Control Plane: ROSA Service Hosted OpenShift Version: 4.16.46 Channel Group: stable DNS: Not ready AWS Account: 418295695583 AWS Billing Account: 418295695583 API URL: Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: MultiAZ Nodes: - Compute (desired): 3 - Compute (current): 0 Network: - Type: OVNKubernetes - Service CIDR: 172.30.0.0/16 - Machine CIDR: 10.0.0.0/16 - Pod CIDR: 10.128.0.0/14 - Host Prefix: /23 - Subnets: subnet-001fc23497e4a3aeb, subnet-00ffba09365a434bc, subnet-074cbf0329958194a, subnet-0689cd077699b690a, subnet-0f9f09e46f74cde64, subnet-033f48892ddbaa09d EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::418295695583:role/ManagedOpenShift-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::418295695583:role/ManagedOpenShift-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::418295695583:role/ManagedOpenShift-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::418295695583:role/rosa-hcp-kube-system-kube-controller-manager - arn:aws:iam::418295695583:role/rosa-hcp-openshift-ingress-operator-cloud-credentials - arn:aws:iam::418295695583:role/rosa-hcp-openshift-cluster-csi-drivers-ebs-cloud-credentials - arn:aws:iam::418295695583:role/rosa-hcp-openshift-cloud-network-config-controller-cloud-credent - arn:aws:iam::418295695583:role/rosa-hcp-openshift-image-registry-installer-cloud-credentials - arn:aws:iam::418295695583:role/rosa-hcp-kube-system-capa-controller-manager - arn:aws:iam::418295695583:role/rosa-hcp-kube-system-control-plane-operator - arn:aws:iam::418295695583:role/rosa-hcp-kube-system-kms-provider Managed Policies: Yes State: waiting (Waiting for user action) Private: No Delete Protection: Disabled Created: Sep 4 2025 05:36:54 UTC User Workload Monitoring: Enabled Details Page: https://console.redhat.com/openshift/details/s/32DnsAKoSJmtKo4R1VcAhQSAyoh OIDC Endpoint URL: https://oidc.op1.openshiftapps.com/2du11g36ejmoo4624pofphlrgf4r9tf3 (Managed) Etcd Encryption: Disabled Audit Log Forwarding: Disabled External Authentication: Disabled Zero Egress: Disabled INFO: Preparing to create operator roles. INFO: Operator Roles already exists INFO: Preparing to create OIDC Provider. INFO: OIDC provider already exists INFO: To determine when your cluster is Ready, run 'rosa describe cluster -c kx-6c470fb64a'. INFO: To watch your cluster installation logs, run 'rosa logs install -c kx-6c470fb64a --watch'. INFO: Track the progress of the cluster creation... W: Region flag will be removed from this command in future versions INFO: Cluster 'kx-6c470fb64a' is in waiting state waiting for installation to begin. Logs will show up within 5 minutes 0001-01-01 00:00:00 +0000 UTC hostedclusters kx-6c470fb64a Version 2025-09-04 05:41:11 +0000 UTC hostedclusters kx-6c470fb64a ValidAWSIdentityProvider StatusUnknown 2025-09-04 05:41:12 +0000 UTC certificates cluster-api-cert Issuing certificate as Secret does not exist 2025-09-04 05:41:12 +0000 UTC certificates cluster-api-cert Issuing certificate as Secret does not exist 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a configuration is invalid: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a ValidConfiguration condition is false: NamedCertificates get secret: Invalid value: "cluster-api-cert": Secret "cluster-api-cert" not found 0001-01-01 00:00:00 +0000 UTC hostedclusters kx-6c470fb64a Version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Waiting for hosted control plane kubeconfig to be created 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is at expected version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment not found 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:42:38 +0000 UTC certificates cluster-api-cert Certificate is up to date and has not expired 2025-09-04 05:42:42 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:44 +0000 UTC hostedclusters kx-6c470fb64a Required platform credentials are found 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a AWS KMS is not configured 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a capi-provider deployment has 2 unavailable replicas 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a lookup api.kx-6c470fb64a.fnn9.p3.openshiftapps.com on 172.30.0.10:53: no such host 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a Waiting for etcd to reach quorum 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a Kube APIServer deployment not found 2025-09-04 05:42:54 +0000 UTC hostedclusters kx-6c470fb64a OIDC configuration is valid 2025-09-04 05:42:55 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation completed successfully 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:20 +0000 UTC hostedclusters kx-6c470fb64a WebIdentityErr 0001-01-01 00:00:00 +0000 UTC hostedclusters kx-6c470fb64a Version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is at expected version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Get "https://a7f9050824b2e4d68b95ba5f0b9ba382-f5f61cd114197435.elb.us-east-1.amazonaws.com:443/healthz": dial tcp 44.209.210.145:443: i/o timeout (Client.Timeout exceeded while awaiting headers) 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:42:38 +0000 UTC certificates cluster-api-cert Certificate is up to date and has not expired 2025-09-04 05:42:42 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:44 +0000 UTC hostedclusters kx-6c470fb64a Required platform credentials are found 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a AWS KMS is not configured 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 2 unavailable replicas, cloud-credential-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 2 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 2 unavailable replicas, redhat-operators-catalog deployment has 2 unavailable replicas, router deployment has 1 unavailable replicas] 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a lookup api.kx-6c470fb64a.fnn9.p3.openshiftapps.com on 172.30.0.10:53: no such host 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:54 +0000 UTC hostedclusters kx-6c470fb64a OIDC configuration is valid 2025-09-04 05:42:55 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation completed successfully 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:45 +0000 UTC hostedclusters kx-6c470fb64a EtcdAvailable QuorumAvailable 2025-09-04 05:44:09 +0000 UTC hostedclusters kx-6c470fb64a Kube APIServer deployment is available 2025-09-04 05:44:20 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:44:28 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:44:34 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment is available 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Payload loaded version="4.16.46" image="quay.io/openshift-release-dev/ocp-release@sha256:780ccd5a4423950437914126ce959bab4cf9e422ca5167609b32e06d31280d8a" architecture="Multi" 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a ClusterVersionAvailable FromClusterVersion 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Unable to apply 4.16.46: an unknown error has occurred: MultipleErrors 2025-09-04 05:45:44 +0000 UTC hostedclusters kx-6c470fb64a Multiple errors are preventing progress: * Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, network, node-tuning, openshift-samples, service-ca are not available * Could not update imagestream "openshift/driver-toolkit" (441 of 624): resource may have been deleted * Could not update operatorgroup "openshift-monitoring/openshift-cluster-monitoring" (554 of 624): resource may have been deleted * Could not update role "openshift-authentication/prometheus-k8s" (541 of 624): resource may have been deleted * Could not update role "openshift-console-operator/prometheus-k8s" (583 of 624): resource may have been deleted * Could not update role "openshift-console/prometheus-k8s" (587 of 624): resource may have been deleted * Could not update role "openshift-ingress-operator/prometheus-k8s" (594 of 624): resource may have been deleted * Could not update role "openshift-kube-apiserver-operator/prometheus-k8s" (598 of 624): resource may have been deleted 0001-01-01 00:00:00 +0000 UTC hostedclusters kx-6c470fb64a Version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is at expected version 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation active on resource 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a Release image is valid 2025-09-04 05:41:14 +0000 UTC hostedclusters kx-6c470fb64a HostedCluster is supported by operator configuration 2025-09-04 05:42:38 +0000 UTC certificates cluster-api-cert Certificate is up to date and has not expired 2025-09-04 05:42:42 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:44 +0000 UTC hostedclusters kx-6c470fb64a Required platform credentials are found 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a Configuration passes validation 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a AWS KMS is not configured 2025-09-04 05:42:51 +0000 UTC hostedclusters kx-6c470fb64a lookup api.kx-6c470fb64a.fnn9.p3.openshiftapps.com on 172.30.0.10:53: no such host 2025-09-04 05:42:54 +0000 UTC hostedclusters kx-6c470fb64a OIDC configuration is valid 2025-09-04 05:42:55 +0000 UTC hostedclusters kx-6c470fb64a Reconciliation completed successfully 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:15 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:43:45 +0000 UTC hostedclusters kx-6c470fb64a EtcdAvailable QuorumAvailable 2025-09-04 05:44:09 +0000 UTC hostedclusters kx-6c470fb64a Kube APIServer deployment is available 2025-09-04 05:44:20 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:44:28 +0000 UTC hostedclusters kx-6c470fb64a All is well 2025-09-04 05:44:34 +0000 UTC hostedclusters kx-6c470fb64a Ignition server deployment is available 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Payload loaded version="4.16.46" image="quay.io/openshift-release-dev/ocp-release@sha256:780ccd5a4423950437914126ce959bab4cf9e422ca5167609b32e06d31280d8a" architecture="Multi" 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a ClusterVersionAvailable FromClusterVersion 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Unable to apply 4.16.46: some cluster operators are not available 2025-09-04 05:45:08 +0000 UTC hostedclusters kx-6c470fb64a Condition not found in the CVO. 2025-09-04 05:45:44 +0000 UTC hostedclusters kx-6c470fb64a Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, node-tuning, openshift-samples, service-ca, storage are not available 2025-09-04 05:46:04 +0000 UTC hostedclusters kx-6c470fb64a The hosted control plane is available INFO: Cluster 'kx-6c470fb64a' is now ready INFO: ROSA with HCP cluster is ready, create a cluster admin account for accessing the cluster INFO: Storing login command... INFO: Check if it's able to login to OCP cluster... Retried 1 times... Retried 2 times... INFO: Check if apiserver is ready... Waiting for cluster operators to be accessible for 2m... NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 4m37s dns 4.16.46 False False True 4m39s DNS "default" is unavailable. image-registry False True True 4m14s Available: The deployment does not have available replicas... ingress False True True 4m23s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights kube-apiserver 4.16.46 True False False 4m31s kube-controller-manager 4.16.46 True False False 4m31s kube-scheduler 4.16.46 True False False 4m31s kube-storage-version-migrator monitoring network 4.16.46 True True False 4m19s DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready... node-tuning False True False 4m33s DaemonSet "tuned" has no available Pod(s) openshift-apiserver 4.16.46 True False False 4m31s openshift-controller-manager 4.16.46 True False False 4m31s openshift-samples operator-lifecycle-manager 4.16.46 True False False 4m32s operator-lifecycle-manager-catalog 4.16.46 True False False 4m20s operator-lifecycle-manager-packageserver 4.16.46 True False False 4m31s service-ca storage 4.16.46 False False False 4m31s AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverNodeServiceControllerAvailable: Waiting for the DaemonSet to deploy the CSI Node Service cluster operators to be accessible finished! [INFO] Cluster operators are accessible. Waiting for cluster to be reported as healthy for 60m... NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 4m37s dns 4.16.46 False False True 4m39s DNS "default" is unavailable. image-registry False True True 4m14s Available: The deployment does not have available replicas... ingress False True True 4m23s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights kube-apiserver 4.16.46 True False False 4m31s kube-controller-manager 4.16.46 True False False 4m31s kube-scheduler 4.16.46 True False False 4m31s kube-storage-version-migrator monitoring network 4.16.46 True True False 4m19s DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready... node-tuning False True False 4m33s DaemonSet "tuned" has no available Pod(s) openshift-apiserver 4.16.46 True False False 4m31s openshift-controller-manager 4.16.46 True False False 4m31s openshift-samples operator-lifecycle-manager 4.16.46 True False False 4m32s operator-lifecycle-manager-catalog 4.16.46 True False False 4m20s operator-lifecycle-manager-packageserver 4.16.46 True False False 4m31s service-ca storage 4.16.46 False False False 4m31s AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverNodeServiceControllerAvailable: Waiting for the DaemonSet to deploy the CSI Node Service Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 5m37s dns 4.16.46 False False True 5m39s DNS "default" is unavailable. image-registry False True True 5m14s Available: The deployment does not have available replicas... ingress False True True 5m23s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights kube-apiserver 4.16.46 True False False 5m31s kube-controller-manager 4.16.46 True False False 5m31s kube-scheduler 4.16.46 True False False 5m31s kube-storage-version-migrator monitoring network 4.16.46 True True False 5m19s DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready... node-tuning False True False 5m33s DaemonSet "tuned" has no available Pod(s) openshift-apiserver 4.16.46 True False False 5m31s openshift-controller-manager 4.16.46 True False False 5m31s openshift-samples operator-lifecycle-manager 4.16.46 True False False 5m32s operator-lifecycle-manager-catalog 4.16.46 True False False 5m20s operator-lifecycle-manager-packageserver 4.16.46 True False False 5m31s service-ca storage 4.16.46 False False False 5m31s AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverNodeServiceControllerAvailable: Waiting for the DaemonSet to deploy the CSI Node Service Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 6m38s dns 4.16.46 False False True 6m40s DNS "default" is unavailable. image-registry False True True 6m15s Available: The deployment does not have available replicas... ingress False True True 6m24s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights kube-apiserver 4.16.46 True False False 6m32s kube-controller-manager 4.16.46 True False False 6m32s kube-scheduler 4.16.46 True False False 6m32s kube-storage-version-migrator monitoring network 4.16.46 True True False 6m20s DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready... node-tuning False True False 6m34s DaemonSet "tuned" has no available Pod(s) openshift-apiserver 4.16.46 True False False 6m32s openshift-controller-manager 4.16.46 True False False 6m32s openshift-samples operator-lifecycle-manager 4.16.46 True False False 6m33s operator-lifecycle-manager-catalog 4.16.46 True False False 6m21s operator-lifecycle-manager-packageserver 4.16.46 True False False 6m32s service-ca storage 4.16.46 False False False 6m32s AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverNodeServiceControllerAvailable: Waiting for the DaemonSet to deploy the CSI Node Service Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 7m38s dns 4.16.46 False False True 7m40s DNS "default" is unavailable. image-registry False True True 7m15s Available: The deployment does not have available replicas... ingress False True True 7m24s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights kube-apiserver 4.16.46 True False False 7m32s kube-controller-manager 4.16.46 True False False 7m32s kube-scheduler 4.16.46 True False False 7m32s kube-storage-version-migrator monitoring network 4.16.46 True True False 7m20s DaemonSet "/openshift-multus/multus-additional-cni-plugins" is not available (awaiting 1 nodes)... node-tuning 4.16.46 True True False 6s Waiting for 1/1 Profiles to be applied openshift-apiserver 4.16.46 True False False 7m32s openshift-controller-manager 4.16.46 True False False 7m32s openshift-samples operator-lifecycle-manager 4.16.46 True False False 7m33s operator-lifecycle-manager-catalog 4.16.46 True False False 7m21s operator-lifecycle-manager-packageserver 4.16.46 True False False 7m32s service-ca storage 4.16.46 True False False 2s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.16.46 True False False 8m38s dns 4.16.46 False True True 8m40s DNS "default" is unavailable. image-registry False True True 8m15s Available: The deployment does not have available replicas... ingress False True True 8m24s The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights 4.16.46 True False False 13s kube-apiserver 4.16.46 True False False 8m32s kube-controller-manager 4.16.46 True False False 8m32s kube-scheduler 4.16.46 True False False 8m32s kube-storage-version-migrator 4.16.46 True False False 9s monitoring network 4.16.46 True True False 8m20s DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready node-tuning 4.16.46 True False False 45s openshift-apiserver 4.16.46 True False False 8m32s openshift-controller-manager 4.16.46 True False False 8m32s openshift-samples operator-lifecycle-manager 4.16.46 True False False 8m33s operator-lifecycle-manager-catalog 4.16.46 True False False 8m21s operator-lifecycle-manager-packageserver 4.16.46 True False False 8m32s service-ca 4.16.46 True False False 11s storage 4.16.46 True False False 62s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True True False 30s SyncLoopRefreshProgressing: working toward version 4.16.46, 1 replicas available csi-snapshot-controller 4.16.46 True False False 9m38s dns 4.16.46 True False False 29s image-registry 4.16.46 True False False 26s ingress 4.16.46 True False False 15s insights 4.16.46 True False False 73s kube-apiserver 4.16.46 True False False 9m32s kube-controller-manager 4.16.46 True False False 9m32s kube-scheduler 4.16.46 True False False 9m32s kube-storage-version-migrator 4.16.46 True False False 69s monitoring Unknown True Unknown 45s Rolling out the stack. network 4.16.46 True False False 9m20s node-tuning 4.16.46 True False False 105s openshift-apiserver 4.16.46 True False False 9m32s openshift-controller-manager 4.16.46 True False False 9m32s openshift-samples 4.16.46 True False False 5s operator-lifecycle-manager 4.16.46 True False False 9m33s operator-lifecycle-manager-catalog 4.16.46 True False False 9m21s operator-lifecycle-manager-packageserver 4.16.46 True False False 9m32s service-ca 4.16.46 True False False 71s storage 4.16.46 True False False 2m2s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True False False 90s csi-snapshot-controller 4.16.46 True False False 10m dns 4.16.46 True False False 89s image-registry 4.16.46 True False False 86s ingress 4.16.46 True False False 75s insights 4.16.46 True False False 2m13s kube-apiserver 4.16.46 True False False 10m kube-controller-manager 4.16.46 True False False 10m kube-scheduler 4.16.46 True False False 10m kube-storage-version-migrator 4.16.46 True False False 2m9s monitoring Unknown True Unknown 105s Rolling out the stack. network 4.16.46 True False False 10m node-tuning 4.16.46 True False False 2m45s openshift-apiserver 4.16.46 True False False 10m openshift-controller-manager 4.16.46 True False False 10m openshift-samples 4.16.46 True False False 65s operator-lifecycle-manager 4.16.46 True False False 10m operator-lifecycle-manager-catalog 4.16.46 True False False 10m operator-lifecycle-manager-packageserver 4.16.46 True False False 10m service-ca 4.16.46 True False False 2m11s storage 4.16.46 True False False 3m2s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True False False 2m31s csi-snapshot-controller 4.16.46 True False False 11m dns 4.16.46 True False False 2m30s image-registry 4.16.46 True False False 2m27s ingress 4.16.46 True False False 2m16s insights 4.16.46 True False False 3m14s kube-apiserver 4.16.46 True False False 11m kube-controller-manager 4.16.46 True False False 11m kube-scheduler 4.16.46 True False False 11m kube-storage-version-migrator 4.16.46 True False False 3m10s monitoring Unknown True Unknown 2m46s Rolling out the stack. network 4.16.46 True False False 11m node-tuning 4.16.46 True False False 3m46s openshift-apiserver 4.16.46 True False False 11m openshift-controller-manager 4.16.46 True False False 11m openshift-samples 4.16.46 True False False 2m6s operator-lifecycle-manager 4.16.46 True False False 11m operator-lifecycle-manager-catalog 4.16.46 True False False 11m operator-lifecycle-manager-packageserver 4.16.46 True False False 11m service-ca 4.16.46 True False False 3m12s storage 4.16.46 True False False 4m3s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True False False 3m31s csi-snapshot-controller 4.16.46 True False False 12m dns 4.16.46 True False False 3m30s image-registry 4.16.46 True False False 3m27s ingress 4.16.46 True False False 3m16s insights 4.16.46 True False False 4m14s kube-apiserver 4.16.46 True False False 12m kube-controller-manager 4.16.46 True False False 12m kube-scheduler 4.16.46 True False False 12m kube-storage-version-migrator 4.16.46 True False False 4m10s monitoring 4.16.46 True False False 15s network 4.16.46 True False False 12m node-tuning 4.16.46 True False False 4m46s openshift-apiserver 4.16.46 True False False 12m openshift-controller-manager 4.16.46 True False False 12m openshift-samples 4.16.46 True False False 3m6s operator-lifecycle-manager 4.16.46 True False False 12m operator-lifecycle-manager-catalog 4.16.46 True False False 12m operator-lifecycle-manager-packageserver 4.16.46 True False False 12m service-ca 4.16.46 True False False 4m12s storage 4.16.46 True False False 5m3s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True False False 4m31s csi-snapshot-controller 4.16.46 True False False 13m dns 4.16.46 True False False 4m30s image-registry 4.16.46 True False False 4m27s ingress 4.16.46 True False False 4m16s insights 4.16.46 True False False 5m14s kube-apiserver 4.16.46 True False False 13m kube-controller-manager 4.16.46 True False False 13m kube-scheduler 4.16.46 True False False 13m kube-storage-version-migrator 4.16.46 True False False 5m10s monitoring 4.16.46 True False False 75s network 4.16.46 True False False 13m node-tuning 4.16.46 True False False 5m46s openshift-apiserver 4.16.46 True False False 13m openshift-controller-manager 4.16.46 True False False 13m openshift-samples 4.16.46 True False False 4m6s operator-lifecycle-manager 4.16.46 True False False 13m operator-lifecycle-manager-catalog 4.16.46 True False False 13m operator-lifecycle-manager-packageserver 4.16.46 True False False 13m service-ca 4.16.46 True False False 5m12s storage 4.16.46 True False False 6m3s Waiting for cluster to be reported as healthy... Trying again in 60s NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.16.46 True False False 5m31s csi-snapshot-controller 4.16.46 True False False 14m dns 4.16.46 True False False 5m30s image-registry 4.16.46 True False False 5m27s ingress 4.16.46 True False False 5m16s insights 4.16.46 True False False 6m14s kube-apiserver 4.16.46 True False False 14m kube-controller-manager 4.16.46 True False False 14m kube-scheduler 4.16.46 True False False 14m kube-storage-version-migrator 4.16.46 True False False 6m10s monitoring 4.16.46 True False False 2m15s network 4.16.46 True False False 14m node-tuning 4.16.46 True False False 6m46s openshift-apiserver 4.16.46 True False False 14m openshift-controller-manager 4.16.46 True False False 14m openshift-samples 4.16.46 True False False 5m6s operator-lifecycle-manager 4.16.46 True False False 14m operator-lifecycle-manager-catalog 4.16.46 True False False 14m operator-lifecycle-manager-packageserver 4.16.46 True False False 14m service-ca 4.16.46 True False False 6m12s storage 4.16.46 True False False 7m3s Waiting for cluster to be reported as healthy... Trying again in 60s healthy cluster to be reported as healthy finished!