I0216 09:54:51.817712 1 controller.go:80] Starting MinIO Operator I0216 09:54:51.904903 1 main-controller.go:323] Setting up event handlers I0216 09:54:52.205668 1 main-controller.go:556] Using Kubernetes CSR Version: v1 I0216 09:54:52.205703 1 main-controller.go:573] Waiting for STS API to start I0216 09:54:52.205741 1 leaderelection.go:250] attempting to acquire leader lease minio-operator/minio-operator-lock... I0216 09:54:52.205744 1 main-controller.go:409] Starting STS API server I0216 09:54:52.210729 1 tls.go:52] Waiting for the sts certificates secret to be issued I0216 09:54:52.304396 1 leaderelection.go:260] successfully acquired lease minio-operator/minio-operator-lock I0216 09:54:52.304412 1 main-controller.go:605] minio-operator-594b66f6f9-75br9: I am the leader, applying leader labels on myself I0216 09:54:52.304603 1 main-controller.go:442] Waiting for Upgrade Server to start I0216 09:54:52.304644 1 main-controller.go:446] Starting Tenant controller I0216 09:54:52.304845 1 main-controller.go:449] Waiting for informer caches to sync I0216 09:54:52.304818 1 main-controller.go:398] Starting HTTP Upgrade Tenant Image server I0216 09:54:52.505573 1 main-controller.go:460] Starting workers and Job workers I0216 09:54:52.505600 1 main-controller.go:496] Console TLS is not enabled I0216 09:54:52.505615 1 main-controller.go:505] STS is enabled, starting API certificate setup I0216 09:54:52.604325 1 tls.go:125] sts-tls TLS secret not found: secrets "sts-tls" not found I0216 09:54:52.729004 1 csr.go:181] Start polling for certificate of csr/sts-minio-operator-csr, every 5s, timeout after 20m0s I0216 09:54:57.733803 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I0216 09:54:57.741686 1 tls.go:128] Waiting for the sts certificates to be issued waiting for sts cert I0216 09:55:00.876422 1 minio.go:308] Generating private key I0216 09:55:00.876499 1 minio.go:321] Generating CSR with CN=*.storage-hl.tekton-results.svc.cluster.local I0216 09:55:00.891644 1 csr.go:181] Start polling for certificate of csr/storage-tekton-results-csr, every 5s, timeout after 20m0s I0216 09:55:00.891687 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"28888", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created I0216 09:55:05.899333 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I0216 09:55:10.904538 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29180", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created I0216 09:55:10.914210 1 status.go:55] Hit conflict issue, getting latest version of tenant I0216 09:55:10.949146 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29186", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created I0216 09:55:10.958605 1 status.go:55] Hit conflict issue, getting latest version of tenant I0216 09:55:10.982701 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29192", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created I0216 09:55:11.052711 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29178", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created I0216 09:55:11.104794 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29178", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created I0216 09:55:11.872522 1 status.go:89] Hit conflict issue, getting latest version of tenant I0216 09:55:11.890094 1 main-controller.go:958] Detected we are updating a legacy tenant deployment I0216 09:55:11.903271 1 main-controller.go:997] 'tekton-results/storage': Deploying pool pool-0 I0216 09:55:11.928001 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29215", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created I0216 09:55:13.062691 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:13.269322 1 pdb.go:191] PodDisruptionBudget: v1 I0216 09:55:15.028584 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:15.047574 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:15.070694 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:15.906279 1 minio-services.go:164] Headless Services don't match: service ports don't match I0216 09:55:15.917253 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"5614ae91-3507-44d2-a534-58758afb90d5", APIVersion:"minio.min.io/v2", ResourceVersion:"29406", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated I0216 09:55:18.639338 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:19.726086 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:55:22.749050 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.96.70:443: connect: connection refused I0216 09:57:52.623058 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0216 09:57:52.623084 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0216 09:57:52.628691 1 tenants.go:134] there are secret(s) for the operator I0216 09:57:52.628705 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628712 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628717 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628722 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628726 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628731 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628736 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0216 09:57:52.628741 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates