I1006 06:40:10.561854 1 controller.go:80] Starting MinIO Operator I1006 06:40:10.651624 1 main-controller.go:323] Setting up event handlers I1006 06:40:10.758598 1 main-controller.go:556] Using Kubernetes CSR Version: v1 I1006 06:40:10.758624 1 main-controller.go:573] Waiting for STS API to start I1006 06:40:10.758655 1 leaderelection.go:250] attempting to acquire leader lease minio-operator/minio-operator-lock... I1006 06:40:10.758666 1 main-controller.go:409] Starting STS API server I1006 06:40:10.951621 1 leaderelection.go:260] successfully acquired lease minio-operator/minio-operator-lock I1006 06:40:10.951795 1 tls.go:52] Waiting for the sts certificates secret to be issued I1006 06:40:10.951877 1 main-controller.go:605] minio-operator-74b9cc98cb-95kzk: I am the leader, applying leader labels on myself I1006 06:40:10.952018 1 main-controller.go:442] Waiting for Upgrade Server to start I1006 06:40:10.952043 1 main-controller.go:446] Starting Tenant controller I1006 06:40:10.952060 1 main-controller.go:449] Waiting for informer caches to sync I1006 06:40:10.952099 1 main-controller.go:398] Starting HTTP Upgrade Tenant Image server I1006 06:40:11.252386 1 main-controller.go:460] Starting workers and Job workers I1006 06:40:11.252413 1 main-controller.go:496] Console TLS is not enabled I1006 06:40:11.252430 1 main-controller.go:505] STS is enabled, starting API certificate setup I1006 06:40:11.281461 1 monitoring.go:92] 'openshift-pipelines/storage' no pool is initialized I1006 06:40:11.363983 1 tls.go:125] sts-tls TLS secret not found: secrets "sts-tls" not found I1006 06:40:11.663338 1 csr.go:181] Start polling for certificate of csr/sts-minio-operator-csr, every 5s, timeout after 20m0s I1006 06:40:15.749064 1 minio.go:308] Generating private key I1006 06:40:15.749132 1 minio.go:321] Generating CSR with CN=*.storage-hl.openshift-pipelines.svc.cluster.local I1006 06:40:15.767410 1 csr.go:181] Start polling for certificate of csr/storage-openshift-pipelines-csr, every 5s, timeout after 20m0s I1006 06:40:15.767450 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27125", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created I1006 06:40:16.668594 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I1006 06:40:16.675381 1 tls.go:128] Waiting for the sts certificates to be issued waiting for sts cert I1006 06:40:20.773328 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I1006 06:40:25.852776 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27753", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created I1006 06:40:25.869514 1 status.go:55] Hit conflict issue, getting latest version of tenant I1006 06:40:25.938657 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27767", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created I1006 06:40:25.952666 1 status.go:55] Hit conflict issue, getting latest version of tenant I1006 06:40:25.991220 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27779", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created I1006 06:40:26.011058 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27125", FieldPath:""}): type: 'Normal' reason: 'SACreated' Service Account Created I1006 06:40:26.137705 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27125", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created I1006 06:40:26.695035 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27125", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created I1006 06:40:27.767492 1 status.go:89] Hit conflict issue, getting latest version of tenant I1006 06:40:27.997873 1 main-controller.go:958] Detected we are updating a legacy tenant deployment I1006 06:40:28.131897 1 main-controller.go:997] 'openshift-pipelines/storage': Deploying pool pool-0 I1006 06:40:28.237051 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27955", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created I1006 06:40:28.519969 1 pdb.go:191] PodDisruptionBudget: v1 I1006 06:40:28.924428 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:30.880202 1 minio-services.go:164] Headless Services don't match: service ports don't match I1006 06:40:30.903492 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-pipelines", Name:"storage", UID:"fee2bd1b-3a7d-4e95-ab32-737e06c0d333", APIVersion:"minio.min.io/v2", ResourceVersion:"27988", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated I1006 06:40:32.952214 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:32.965847 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:32.987403 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:42.870336 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:48.359641 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": dial tcp 172.30.137.219:443: connect: connection refused I1006 06:40:52.431744 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I1006 06:40:52.431783 1 tenants.go:126] Let's check if there is an external cert for the tenant... I1006 06:40:52.446529 1 tenants.go:134] there are secret(s) for the operator I1006 06:40:52.446549 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446558 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446565 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446574 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446581 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446589 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446598 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:40:52.446609 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:41:34.892985 1 minio.go:308] Generating private key I1006 06:41:34.893060 1 minio.go:321] Generating CSR with CN=*.oadp-minio-storage-hl.openshift-adp.svc.cluster.local I1006 06:41:34.910502 1 csr.go:181] Start polling for certificate of csr/oadp-minio-storage-openshift-adp-csr, every 5s, timeout after 20m0s I1006 06:41:34.910566 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30648", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created I1006 06:41:39.915719 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I1006 06:41:44.937486 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30850", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created I1006 06:41:44.947005 1 status.go:55] Hit conflict issue, getting latest version of tenant I1006 06:41:44.983686 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30856", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created I1006 06:41:44.993104 1 status.go:55] Hit conflict issue, getting latest version of tenant I1006 06:41:45.023683 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30862", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created I1006 06:41:45.071676 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30648", FieldPath:""}): type: 'Normal' reason: 'SACreated' Service Account Created I1006 06:41:45.143895 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30648", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created I1006 06:41:45.248381 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30648", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created I1006 06:41:45.483460 1 status.go:89] Hit conflict issue, getting latest version of tenant I1006 06:41:45.498889 1 main-controller.go:958] Detected we are updating a legacy tenant deployment I1006 06:41:45.671705 1 main-controller.go:997] 'openshift-adp/oadp-minio-storage': Deploying pool pool-0 I1006 06:41:45.883191 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30898", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created I1006 06:41:47.073046 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:41:49.016741 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:41:49.034144 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:41:49.054602 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:41:49.932081 1 minio-services.go:164] Headless Services don't match: service ports don't match I1006 06:41:49.940405 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"openshift-adp", Name:"oadp-minio-storage", UID:"23d5d518-825b-40e7-ad8a-2db493ccd65f", APIVersion:"minio.min.io/v2", ResourceVersion:"30948", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated I1006 06:41:57.723029 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:41:58.528099 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:42:00.552085 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": dial tcp 172.30.241.197:443: connect: connection refused I1006 06:43:11.300894 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I1006 06:43:11.300918 1 tenants.go:126] Let's check if there is an external cert for the tenant... I1006 06:43:11.314940 1 tenants.go:134] there are secret(s) for the operator I1006 06:43:11.314957 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314964 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314969 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314974 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314982 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314987 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314992 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.314997 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.327390 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I1006 06:43:11.327411 1 tenants.go:126] Let's check if there is an external cert for the tenant... I1006 06:43:11.413715 1 tenants.go:134] there are secret(s) for the operator I1006 06:43:11.413733 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413739 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413745 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413750 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413755 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413772 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413777 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:43:11.413782 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.301466 1 monitoring.go:122] 'openshift-adp/oadp-minio-storage' Failed to get cluster health: Get "https://minio.openshift-adp.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I1006 06:46:11.301491 1 tenants.go:126] Let's check if there is an external cert for the tenant... I1006 06:46:11.310305 1 tenants.go:134] there are secret(s) for the operator I1006 06:46:11.310320 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310326 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310331 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310336 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310341 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310346 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310352 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.310357 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.322110 1 monitoring.go:122] 'openshift-pipelines/storage' Failed to get cluster health: Get "https://minio.openshift-pipelines.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I1006 06:46:11.322127 1 tenants.go:126] Let's check if there is an external cert for the tenant... I1006 06:46:11.327561 1 tenants.go:134] there are secret(s) for the operator I1006 06:46:11.327573 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327578 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327584 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327589 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327593 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327598 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327603 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I1006 06:46:11.327608 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates