I0809 22:25:35.278520 1 controller.go:80] Starting MinIO Operator I0809 22:25:35.298687 1 main-controller.go:323] Setting up event handlers I0809 22:25:35.676447 1 main-controller.go:556] Using Kubernetes CSR Version: v1 I0809 22:25:35.676480 1 main-controller.go:573] Waiting for STS API to start I0809 22:25:35.676519 1 leaderelection.go:250] attempting to acquire leader lease minio-operator/minio-operator-lock... I0809 22:25:35.676527 1 main-controller.go:409] Starting STS API server I0809 22:25:35.682030 1 tls.go:52] Waiting for the sts certificates secret to be issued I0809 22:25:35.689321 1 leaderelection.go:260] successfully acquired lease minio-operator/minio-operator-lock I0809 22:25:35.689422 1 main-controller.go:605] minio-operator-54646886cb-6mjf4: I am the leader, applying leader labels on myself I0809 22:25:35.689545 1 main-controller.go:442] Waiting for Upgrade Server to start I0809 22:25:35.689549 1 main-controller.go:446] Starting Tenant controller I0809 22:25:35.689551 1 main-controller.go:449] Waiting for informer caches to sync I0809 22:25:35.689564 1 main-controller.go:398] Starting HTTP Upgrade Tenant Image server I0809 22:25:35.876407 1 main-controller.go:460] Starting workers and Job workers I0809 22:25:35.876436 1 main-controller.go:496] Console TLS is not enabled I0809 22:25:35.876449 1 main-controller.go:505] STS is enabled, starting API certificate setup I0809 22:25:35.882004 1 monitoring.go:92] 'tekton-results/storage' no pool is initialized I0809 22:25:35.886462 1 tls.go:125] sts-tls TLS secret not found: secrets "sts-tls" not found I0809 22:25:35.998656 1 csr.go:181] Start polling for certificate of csr/sts-minio-operator-csr, every 5s, timeout after 20m0s I0809 22:25:40.442256 1 minio.go:308] Generating private key I0809 22:25:40.442318 1 minio.go:321] Generating CSR with CN=*.storage-hl.tekton-results.svc.cluster.local I0809 22:25:40.461425 1 csr.go:181] Start polling for certificate of csr/storage-tekton-results-csr, every 5s, timeout after 20m0s I0809 22:25:40.461468 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"25548", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created I0809 22:25:41.004032 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I0809 22:25:41.012070 1 tls.go:128] Waiting for the sts certificates to be issued waiting for sts cert I0809 22:25:45.467774 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate I0809 22:25:50.502767 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"26690", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created I0809 22:25:50.515566 1 status.go:55] Hit conflict issue, getting latest version of tenant I0809 22:25:50.552817 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"26699", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created I0809 22:25:50.563255 1 status.go:55] Hit conflict issue, getting latest version of tenant I0809 22:25:50.595331 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"26705", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created I0809 22:25:50.822378 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"25548", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created I0809 22:25:50.940459 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"25548", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created I0809 22:25:51.656033 1 status.go:89] Hit conflict issue, getting latest version of tenant I0809 22:25:51.677210 1 main-controller.go:958] Detected we are updating a legacy tenant deployment I0809 22:25:51.694186 1 main-controller.go:997] 'tekton-results/storage': Deploying pool pool-0 I0809 22:25:51.727898 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"26828", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created I0809 22:25:52.626529 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:25:53.034172 1 pdb.go:191] PodDisruptionBudget: v1 I0809 22:25:54.911587 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:25:54.933746 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:25:54.949141 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:25:55.514293 1 minio-services.go:164] Headless Services don't match: service ports don't match I0809 22:25:55.520975 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tekton-results", Name:"storage", UID:"9c8a072d-786b-4ab4-9f72-c732946b0912", APIVersion:"minio.min.io/v2", ResourceVersion:"26895", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated I0809 22:26:02.064471 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:26:03.873058 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:26:06.882628 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": dial tcp 172.30.50.83:443: connect: connection refused I0809 22:28:35.900848 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:28:35.900890 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:28:35.907799 1 tenants.go:134] there are secret(s) for the operator I0809 22:28:35.907819 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907826 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907831 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907836 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907841 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907846 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907851 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:28:35.907855 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.901623 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:31:35.901649 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:31:35.908422 1 tenants.go:134] there are secret(s) for the operator I0809 22:31:35.908440 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908447 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908452 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908457 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908462 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908467 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908471 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:31:35.908476 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.902324 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:34:35.902354 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:34:35.909464 1 tenants.go:134] there are secret(s) for the operator I0809 22:34:35.909485 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909492 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909498 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909503 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909507 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909512 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909517 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:34:35.909522 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.902459 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:37:35.902483 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:37:35.909402 1 tenants.go:134] there are secret(s) for the operator I0809 22:37:35.909425 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909433 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909439 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909443 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909448 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909453 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909458 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:37:35.909463 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.901307 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:40:35.901334 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:40:35.908240 1 tenants.go:134] there are secret(s) for the operator I0809 22:40:35.908257 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908263 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908269 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908273 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908280 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908285 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908290 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:40:35.908295 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.901726 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:43:35.901754 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:43:35.908654 1 tenants.go:134] there are secret(s) for the operator I0809 22:43:35.908676 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908682 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908687 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908707 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908714 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908719 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908724 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:43:35.908729 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.900058 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:46:35.900081 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:46:35.906182 1 tenants.go:134] there are secret(s) for the operator I0809 22:46:35.906195 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906201 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906206 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906211 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906216 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906221 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906226 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:46:35.906231 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.899714 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:49:35.899737 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:49:35.905624 1 tenants.go:134] there are secret(s) for the operator I0809 22:49:35.905637 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905643 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905648 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905653 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905658 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905663 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905668 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:49:35.905672 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.900785 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:52:35.900809 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:52:35.907648 1 tenants.go:134] there are secret(s) for the operator I0809 22:52:35.907663 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907669 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907674 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907679 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907684 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907688 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907708 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:52:35.907714 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.900863 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:55:35.900886 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:55:35.907799 1 tenants.go:134] there are secret(s) for the operator I0809 22:55:35.907813 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907819 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907824 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907829 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907834 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907839 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907844 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:55:35.907848 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.901024 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 22:58:35.901047 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 22:58:35.907985 1 tenants.go:134] there are secret(s) for the operator I0809 22:58:35.907999 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908005 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908010 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908016 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908021 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908026 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908031 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 22:58:35.908035 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.901463 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:01:35.901486 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:01:35.908139 1 tenants.go:134] there are secret(s) for the operator I0809 23:01:35.908154 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908160 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908165 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908170 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908175 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908180 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908185 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:01:35.908190 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.901965 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:04:35.901989 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:04:35.908562 1 tenants.go:134] there are secret(s) for the operator I0809 23:04:35.908577 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908584 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908590 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908594 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908599 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908604 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908609 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:04:35.908614 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.900640 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:07:35.900662 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:07:35.907753 1 tenants.go:134] there are secret(s) for the operator I0809 23:07:35.907770 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907776 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907781 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907787 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907792 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907797 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907802 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:07:35.907807 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.900159 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:10:35.900184 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:10:35.907335 1 tenants.go:134] there are secret(s) for the operator I0809 23:10:35.907350 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907357 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907361 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907366 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907371 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907376 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907381 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:10:35.907385 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.900656 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:13:35.900681 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:13:35.907262 1 tenants.go:134] there are secret(s) for the operator I0809 23:13:35.907278 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907284 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907289 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907294 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907298 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907303 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907308 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:13:35.907313 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.900057 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:16:35.900082 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:16:35.906819 1 tenants.go:134] there are secret(s) for the operator I0809 23:16:35.906835 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906845 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906850 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906855 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906860 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906865 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906874 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:16:35.906879 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.901041 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:19:35.901067 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:19:35.907848 1 tenants.go:134] there are secret(s) for the operator I0809 23:19:35.907861 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907868 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907874 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907878 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907883 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907888 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907893 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:19:35.907899 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.901008 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:22:35.901030 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:22:35.907858 1 tenants.go:134] there are secret(s) for the operator I0809 23:22:35.907876 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907882 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907888 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907893 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907898 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907903 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907908 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:22:35.907913 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.899668 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:25:35.899690 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:25:35.906493 1 tenants.go:134] there are secret(s) for the operator I0809 23:25:35.906508 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906514 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906519 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906524 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906532 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906537 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906544 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:25:35.906549 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.903074 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:28:35.903103 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:28:35.910082 1 tenants.go:134] there are secret(s) for the operator I0809 23:28:35.910101 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910108 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910113 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910118 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910123 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910128 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910135 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:28:35.910140 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.901125 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:31:35.901148 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:31:35.907272 1 tenants.go:134] there are secret(s) for the operator I0809 23:31:35.907289 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907295 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907300 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907306 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907314 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907324 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907330 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:31:35.907335 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.902635 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:34:35.902661 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:34:35.909621 1 tenants.go:134] there are secret(s) for the operator I0809 23:34:35.909642 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909651 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909659 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909665 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909670 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909675 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909680 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:34:35.909685 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.901041 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:37:35.901064 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:37:35.907656 1 tenants.go:134] there are secret(s) for the operator I0809 23:37:35.907671 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907677 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907683 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907687 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907708 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907713 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907718 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:37:35.907725 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.899900 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:40:35.899924 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:40:35.906545 1 tenants.go:134] there are secret(s) for the operator I0809 23:40:35.906558 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906564 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906569 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906574 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906579 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906584 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906589 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:40:35.906596 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.901281 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:43:35.901301 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:43:35.908137 1 tenants.go:134] there are secret(s) for the operator I0809 23:43:35.908152 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908158 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908164 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908168 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908173 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908178 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908183 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:43:35.908188 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.901167 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:46:35.901191 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:46:35.907932 1 tenants.go:134] there are secret(s) for the operator I0809 23:46:35.907947 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907954 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907959 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907964 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907969 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907974 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907980 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:46:35.907985 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.901040 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:49:35.901063 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:49:35.908017 1 tenants.go:134] there are secret(s) for the operator I0809 23:49:35.908038 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908044 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908049 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908055 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908060 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908065 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908070 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:49:35.908075 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.904645 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:52:35.904670 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:52:35.911804 1 tenants.go:134] there are secret(s) for the operator I0809 23:52:35.911818 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911825 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911829 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911834 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911839 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911844 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911849 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:52:35.911854 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.900964 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:55:35.900986 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:55:35.907843 1 tenants.go:134] there are secret(s) for the operator I0809 23:55:35.907859 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907865 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907870 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907875 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907880 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907885 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907889 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:55:35.907894 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.900640 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0809 23:58:35.900665 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0809 23:58:35.907413 1 tenants.go:134] there are secret(s) for the operator I0809 23:58:35.907428 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907435 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907440 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907445 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907450 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907455 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907460 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0809 23:58:35.907465 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.901363 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0810 00:01:35.901388 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0810 00:01:35.907978 1 tenants.go:134] there are secret(s) for the operator I0810 00:01:35.907995 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908002 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908007 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908013 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908021 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908029 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908037 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:01:35.908045 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.901574 1 monitoring.go:122] 'tekton-results/storage' Failed to get cluster health: Get "https://minio.tekton-results.svc.cluster.local/minio/health/cluster": tls: failed to verify certificate: x509: certificate signed by unknown authority I0810 00:04:35.901602 1 tenants.go:126] Let's check if there is an external cert for the tenant... I0810 00:04:35.908239 1 tenants.go:134] there are secret(s) for the operator I0810 00:04:35.908258 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908264 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908269 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908274 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908279 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908284 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908289 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates I0810 00:04:35.908294 1 tenants.go:58] No secret found for multi-tenancy architecture of external certificates