=== events in opendatahub (2026-10-30T01:42:55Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 23m Normal Scheduled pod/test-curl-8883-8c5452 Successfully assigned opendatahub/test-curl-8883-8c5452 to ip-10-0-130-5.ec2.internal 54m Normal Scheduled pod/maas-controller-577f596567-p422d Successfully assigned opendatahub/maas-controller-577f596567-p422d to ip-10-0-130-171.ec2.internal 54m Normal Scheduled pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Successfully assigned opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-25kxk to ip-10-0-130-171.ec2.internal 38m Normal Scheduled pod/test-curl-8901-a809d5 Successfully assigned opendatahub/test-curl-8901-a809d5 to ip-10-0-130-171.ec2.internal 40m Normal Scheduled pod/test-curl-8901-649818 Successfully assigned opendatahub/test-curl-8901-649818 to ip-10-0-131-254.ec2.internal 37m Normal Scheduled pod/test-curl-8898-953ee2 Successfully assigned opendatahub/test-curl-8898-953ee2 to ip-10-0-131-254.ec2.internal 37m Normal Scheduled pod/test-curl-8898-58b3b1 Successfully assigned opendatahub/test-curl-8898-58b3b1 to ip-10-0-131-254.ec2.internal 37m Normal Scheduled pod/test-curl-8898-310502 Successfully assigned opendatahub/test-curl-8898-310502 to ip-10-0-131-254.ec2.internal 52m Normal Scheduled pod/ai-gateway-controller-5f85cd9cfc-vz8dz Successfully assigned opendatahub/ai-gateway-controller-5f85cd9cfc-vz8dz to ip-10-0-130-171.ec2.internal 37m Normal Scheduled pod/test-curl-8898-2de7aa Successfully assigned opendatahub/test-curl-8898-2de7aa to ip-10-0-131-254.ec2.internal 24m Normal Scheduled pod/test-curl-8889-73b17f Successfully assigned opendatahub/test-curl-8889-73b17f to ip-10-0-130-5.ec2.internal 45m Normal Scheduled pod/test-curl-8886-e46353 Successfully assigned opendatahub/test-curl-8886-e46353 to ip-10-0-131-254.ec2.internal 45m Normal Scheduled pod/test-curl-8886-3e824c Successfully assigned opendatahub/test-curl-8886-3e824c to ip-10-0-131-254.ec2.internal 23m Normal Scheduled pod/test-curl-8883-fbaa72 Successfully assigned opendatahub/test-curl-8883-fbaa72 to ip-10-0-130-5.ec2.internal 52m Normal Scheduled pod/ai-gateway-controller-5d8c498c66-26f4b Successfully assigned opendatahub/ai-gateway-controller-5d8c498c66-26f4b to ip-10-0-130-171.ec2.internal 58m Normal Scheduled pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Successfully assigned opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc to ip-10-0-130-5.ec2.internal 54m Normal Scheduled pod/maas-controller-78d6f77c9d-jn4xq Successfully assigned opendatahub/maas-controller-78d6f77c9d-jn4xq to ip-10-0-130-171.ec2.internal 13m Normal Scheduled pod/maas-controller-78d6f77c9d-7kkl4 Successfully assigned opendatahub/maas-controller-78d6f77c9d-7kkl4 to ip-10-0-130-171.ec2.internal 59m Normal Scheduled pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Successfully assigned opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q to ip-10-0-130-171.ec2.internal 59m Normal Scheduled pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Successfully assigned opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf to ip-10-0-131-254.ec2.internal 14m Normal Scheduled pod/maas-controller-78d6f77c9d-b79zr Successfully assigned opendatahub/maas-controller-78d6f77c9d-b79zr to ip-10-0-130-171.ec2.internal 57m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh to ip-10-0-130-5.ec2.internal 57m Normal Scheduled pod/kserve-controller-manager-776cc66d47-hp8xp Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-hp8xp to ip-10-0-130-5.ec2.internal 58m Normal Scheduled pod/odh-observability-584f99c788-6d7mf Successfully assigned opendatahub/odh-observability-584f99c788-6d7mf to ip-10-0-130-5.ec2.internal 57m Normal Scheduled pod/odh-model-controller-f87fcf56c-zkgm7 Successfully assigned opendatahub/odh-model-controller-f87fcf56c-zkgm7 to ip-10-0-130-5.ec2.internal 57m Normal Scheduled pod/model-serving-api-75bd89fdfc-x78xr Successfully assigned opendatahub/model-serving-api-75bd89fdfc-x78xr to ip-10-0-130-5.ec2.internal 59m Normal Scheduled pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Successfully assigned opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f to ip-10-0-130-5.ec2.internal 54m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-fnvkh Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh to ip-10-0-130-5.ec2.internal 59m Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 requirements not yet checked 59m Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 one or more requirements couldn't be found 59m Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 all requirements found, attempting install 59m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 waiting for install components to report healthy 59m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8cc7fc69d Created pod: opendatahub-operator-controller-manager-8cc7fc69d-vbxkf 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal AddedInterface pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Add eth0 [10.132.0.27/23] from ovn-kubernetes 59m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-8cc7fc69d from 0 to 3 59m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8cc7fc69d Created pod: opendatahub-operator-controller-manager-8cc7fc69d-wzp4q 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal AddedInterface pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Add eth0 [10.134.0.29/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8cc7fc69d Created pod: opendatahub-operator-controller-manager-8cc7fc69d-7kw5f 59m Normal AddedInterface pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Add eth0 [10.133.0.17/23] from ovn-kubernetes 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.87s (2.87s including waiting). Image size: 273709574 bytes. 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.756s (2.756s including waiting). Image size: 273709574 bytes. 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Container created 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Container started 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Container started 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Container created 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Container created 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Container started 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.861s (2.861s including waiting). Image size: 273709574 bytes. 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Container started 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 297ms (297ms including waiting). Image size: 273709574 bytes. 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-wzp4q Container created 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Container started 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 332ms (332ms including waiting). Image size: 273709574 bytes. 59m Normal Started pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Container started 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Container created 59m Normal Pulled pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 347ms (347ms including waiting). Image size: 273709574 bytes. 59m Normal Created pod/opendatahub-operator-controller-manager-8cc7fc69d-vbxkf Container created 59m Normal Pulling pod/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 59m Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-8cc7fc69d-wzp4q_b7132c90-26ee-4eec-b96b-98aaf759e942 became leader 59m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 install strategy completed with no errors 58m Normal CertificateIssued certificaterequest/odh-observability-webhook-cert-1 Certificate fetched from issuer successfully 58m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 58m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 58m Normal SuccessfulCreate replicaset/odh-observability-584f99c788 Created pod: odh-observability-584f99c788-6d7mf 58m Normal Issuing certificate/odh-observability-webhook-cert The certificate has been successfully issued 58m Normal Requested certificate/odh-observability-webhook-cert Created new CertificateRequest resource "odh-observability-webhook-cert-1" 58m Normal Generated certificate/odh-observability-webhook-cert Stored new private key in temporary Secret resource "odh-observability-webhook-cert-xjgh2" 58m Normal Issuing certificate/odh-observability-webhook-cert Issuing certificate as Secret does not exist 58m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 58m Warning FailedMount pod/odh-observability-584f99c788-6d7mf MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 58m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 58m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 58m Normal ScalingReplicaSet deployment/odh-observability Scaled up replica set odh-observability-584f99c788 from 0 to 1 58m Normal cert-manager.io certificaterequest/odh-observability-webhook-cert-1 Certificate request has been approved by cert-manager.io 58m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 58m Normal AddedInterface pod/odh-observability-584f99c788-6d7mf Add eth0 [10.133.0.19/23] from ovn-kubernetes 58m Normal Pulling pod/odh-observability-584f99c788-6d7mf Pulling image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" 58m Normal Pulled pod/odh-observability-584f99c788-6d7mf Successfully pulled image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" in 2.306s (2.306s including waiting). Image size: 154660917 bytes. 58m Normal Created pod/odh-observability-584f99c788-6d7mf Container created 58m Normal Started pod/odh-observability-584f99c788-6d7mf Container started 58m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-5cfb77bf8c Created pod: kserve-module-controller-manager-5cfb77bf8c-gkxfc 58m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-5cfb77bf8c from 0 to 1 58m Normal Pulling pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 58m Normal AddedInterface pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Add eth0 [10.133.0.20/23] from ovn-kubernetes 58m Normal Created pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Container created 58m Normal Pulled pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.24s (2.24s including waiting). Image size: 195729000 bytes. 58m Normal Started pod/kserve-module-controller-manager-5cfb77bf8c-gkxfc Container started 58m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-5cfb77bf8c-gkxfc_f99cbf7d-5e47-4221-83a7-b4dbaf9191fd became leader 57m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-x78xr 57m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Add eth0 [10.133.0.22/23] from ovn-kubernetes 57m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 57m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 57m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-hp8xp 57m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-xz2hh 57m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 57m Normal AddedInterface pod/odh-model-controller-f87fcf56c-zkgm7 Add eth0 [10.133.0.24/23] from ovn-kubernetes 57m Normal Pulling pod/odh-model-controller-f87fcf56c-zkgm7 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 57m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 57m Normal AddedInterface pod/model-serving-api-75bd89fdfc-x78xr Add eth0 [10.133.0.23/23] from ovn-kubernetes 57m Normal Pulling pod/model-serving-api-75bd89fdfc-x78xr Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 57m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-f87fcf56c from 0 to 1 57m Normal SuccessfulCreate replicaset/odh-model-controller-f87fcf56c Created pod: odh-model-controller-f87fcf56c-zkgm7 57m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-hp8xp Add eth0 [10.133.0.21/23] from ovn-kubernetes 57m Normal Pulling pod/kserve-controller-manager-776cc66d47-hp8xp Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 57m Normal Pulled pod/kserve-controller-manager-776cc66d47-hp8xp Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.865s (2.865s including waiting). Image size: 247676581 bytes. 57m Normal Created pod/kserve-controller-manager-776cc66d47-hp8xp Container created 57m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Container started 57m Normal Started pod/model-serving-api-75bd89fdfc-x78xr Container started 57m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Container created 57m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.752s (2.752s including waiting). Image size: 235100981 bytes. 57m Normal Started pod/kserve-controller-manager-776cc66d47-hp8xp Container started 57m Normal Created pod/model-serving-api-75bd89fdfc-x78xr Container created 57m Normal Pulled pod/model-serving-api-75bd89fdfc-x78xr Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.641s (2.641s including waiting). Image size: 184925761 bytes. 57m Normal Created pod/odh-model-controller-f87fcf56c-zkgm7 Container created 57m Normal Started pod/odh-model-controller-f87fcf56c-zkgm7 Container started 57m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-hp8xp_8575425a-9477-4509-9c3a-8462dfde3c51 became leader 57m Normal Pulled pod/odh-model-controller-f87fcf56c-zkgm7 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.473s (3.473s including waiting). Image size: 236755567 bytes. 57m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-f87fcf56c-zkgm7_6a2c6d83-2c09-4f41-8c64-8cef86652160 became leader 57m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-xz2hh_c2b63604-3a04-4744-a788-160e2d052ba3 became leader 54m Normal SuccessfulCreate replicaset/maas-controller-78d6f77c9d Created pod: maas-controller-78d6f77c9d-jn4xq 54m Warning FailedMount pod/maas-controller-577f596567-p422d MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 54m Normal SuccessfulCreate replicaset/maas-controller-577f596567 Created pod: maas-controller-577f596567-p422d 54m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-577f596567 from 0 to 1 54m Normal Pulling pod/maas-controller-577f596567-p422d Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 54m Normal AddedInterface pod/maas-controller-78d6f77c9d-jn4xq Add eth0 [10.132.0.43/23] from ovn-kubernetes 54m Normal Pulling pod/maas-controller-78d6f77c9d-jn4xq Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 54m Normal AddedInterface pod/maas-controller-577f596567-p422d Add eth0 [10.132.0.42/23] from ovn-kubernetes 54m Normal Started pod/maas-controller-577f596567-p422d Container started 54m Normal Pulled pod/maas-controller-577f596567-p422d Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.622s (2.622s including waiting). Image size: 199691795 bytes. 54m Normal Created pod/maas-controller-577f596567-p422d Container created 54m Normal Pulled pod/maas-controller-78d6f77c9d-jn4xq Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.719s (2.719s including waiting). Image size: 199691795 bytes. 54m Normal Started pod/maas-controller-78d6f77c9d-jn4xq Container started 54m Normal Created pod/maas-controller-78d6f77c9d-jn4xq Container created 54m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78d6f77c9d-jn4xq_ebd4d88b-8dea-4e23-b70d-9b5f67175c9c became leader 54m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-7d94f55f4b Created pod: usage-logs-tenancy-proxy-7d94f55f4b-25kxk 54m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-7d94f55f4b from 0 to 1 54m Warning FailedMount pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 54m Normal Pulling pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 54m Normal AddedInterface pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Add eth0 [10.132.0.44/23] from ovn-kubernetes 54m Normal Killing pod/maas-controller-577f596567-p422d Stopping container manager 54m Normal SuccessfulDelete replicaset/maas-controller-577f596567 Deleted pod: maas-controller-577f596567-p422d 54m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-577f596567 from 1 to 0 54m Normal Pulled pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 9.835s (9.835s including waiting). Image size: 1089502687 bytes. 54m Normal Created pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Container created 54m Normal Started pod/usage-logs-tenancy-proxy-7d94f55f4b-25kxk Container started 54m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-fnvkh 54m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 54m Normal Created pod/llmisvc-controller-manager-6466c554d8-fnvkh Container created 54m Normal Started pod/llmisvc-controller-manager-6466c554d8-fnvkh Container started 54m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-fnvkh Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 259ms (259ms including waiting). Image size: 235100981 bytes. 54m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-fnvkh Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 54m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-fnvkh Add eth0 [10.133.0.27/23] from ovn-kubernetes 53m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-xz2hh 53m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-xz2hh Stopping container manager 53m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 53m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-fnvkh_a7e97282-a61a-43eb-bec8-3cd533f2d3bd became leader 52m Normal SuccessfulCreate replicaset/ai-gateway-controller-5f85cd9cfc Created pod: ai-gateway-controller-5f85cd9cfc-vz8dz 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5f85cd9cfc from 0 to 1 52m Normal Pulling pod/ai-gateway-controller-5f85cd9cfc-vz8dz Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 52m Normal AddedInterface pod/ai-gateway-controller-5f85cd9cfc-vz8dz Add eth0 [10.132.0.49/23] from ovn-kubernetes 52m Normal Pulled pod/ai-gateway-controller-5f85cd9cfc-vz8dz Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 2.24s (2.24s including waiting). Image size: 183900584 bytes. 52m Normal Started pod/ai-gateway-controller-5f85cd9cfc-vz8dz Container started 52m Normal Created pod/ai-gateway-controller-5f85cd9cfc-vz8dz Container created 52m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5f85cd9cfc-vz8dz_3dd4ab89-21ca-493f-a7ec-b5b37fe34b75 became leader 52m Normal SuccessfulCreate replicaset/ai-gateway-controller-5d8c498c66 Created pod: ai-gateway-controller-5d8c498c66-26f4b 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5d8c498c66 from 0 to 1 52m Normal Created pod/ai-gateway-controller-5d8c498c66-26f4b Container created 52m Normal AddedInterface pod/ai-gateway-controller-5d8c498c66-26f4b Add eth0 [10.132.0.53/23] from ovn-kubernetes 52m Normal Pulling pod/ai-gateway-controller-5d8c498c66-26f4b Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 52m Normal Pulled pod/ai-gateway-controller-5d8c498c66-26f4b Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 368ms (368ms including waiting). Image size: 183900584 bytes. 52m Normal Started pod/ai-gateway-controller-5d8c498c66-26f4b Container started 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5f85cd9cfc from 1 to 0 52m Normal SuccessfulDelete replicaset/ai-gateway-controller-5f85cd9cfc Deleted pod: ai-gateway-controller-5f85cd9cfc-vz8dz 52m Normal Killing pod/ai-gateway-controller-5f85cd9cfc-vz8dz Stopping container manager 51m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5d8c498c66-26f4b_8eab4079-5b56-4704-8552-d74264493a09 became leader 45m Normal AddedInterface pod/test-curl-8886-3e824c Add eth0 [10.134.0.84/23] from ovn-kubernetes 45m Normal Pulling pod/test-curl-8886-3e824c Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 45m Normal Started pod/test-curl-8886-3e824c Container started 45m Normal Created pod/test-curl-8886-3e824c Container created 45m Normal Pulled pod/test-curl-8886-3e824c Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 6.798s (6.798s including waiting). Image size: 106100986 bytes. 45m Normal Pulled pod/test-curl-8886-e46353 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/test-curl-8886-e46353 Add eth0 [10.134.0.85/23] from ovn-kubernetes 45m Normal Started pod/test-curl-8886-e46353 Container started 45m Normal Created pod/test-curl-8886-e46353 Container created 45m Normal Killing pod/test-curl-8886-3e824c Stopping container test-curl-8886-3e824c 45m Normal Killing pod/test-curl-8886-e46353 Stopping container test-curl-8886-e46353 40m Normal Pulled pod/test-curl-8901-649818 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/test-curl-8901-649818 Add eth0 [10.134.0.96/23] from ovn-kubernetes 40m Normal Created pod/test-curl-8901-649818 Container created 40m Normal Started pod/test-curl-8901-649818 Container started 40m Normal Killing pod/test-curl-8901-649818 Stopping container test-curl-8901-649818 38m Normal AddedInterface pod/test-curl-8901-a809d5 Add eth0 [10.132.0.209/23] from ovn-kubernetes 38m Normal Pulling pod/test-curl-8901-a809d5 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 38m Normal Pulled pod/test-curl-8901-a809d5 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.546s (2.546s including waiting). Image size: 106100986 bytes. 38m Normal Created pod/test-curl-8901-a809d5 Container created 38m Normal Started pod/test-curl-8901-a809d5 Container started 38m Normal Killing pod/test-curl-8901-a809d5 Stopping container test-curl-8901-a809d5 37m Normal Pulled pod/test-curl-8898-58b3b1 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 37m Normal Started pod/test-curl-8898-58b3b1 Container started 37m Normal Created pod/test-curl-8898-58b3b1 Container created 37m Normal AddedInterface pod/test-curl-8898-58b3b1 Add eth0 [10.134.0.104/23] from ovn-kubernetes 37m Normal AddedInterface pod/test-curl-8898-310502 Add eth0 [10.134.0.105/23] from ovn-kubernetes 37m Normal Started pod/test-curl-8898-310502 Container started 37m Normal Created pod/test-curl-8898-310502 Container created 37m Normal Pulled pod/test-curl-8898-310502 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 37m Normal Killing pod/test-curl-8898-58b3b1 Stopping container test-curl-8898-58b3b1 37m Normal Started pod/test-curl-8898-953ee2 Container started 37m Normal Created pod/test-curl-8898-953ee2 Container created 37m Normal Pulled pod/test-curl-8898-953ee2 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/test-curl-8898-953ee2 Add eth0 [10.134.0.106/23] from ovn-kubernetes 37m Normal Killing pod/test-curl-8898-310502 Stopping container test-curl-8898-310502 37m Normal Pulled pod/test-curl-8898-2de7aa Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/test-curl-8898-2de7aa Add eth0 [10.134.0.107/23] from ovn-kubernetes 37m Normal Started pod/test-curl-8898-2de7aa Container started 37m Normal Created pod/test-curl-8898-2de7aa Container created 37m Normal Killing pod/test-curl-8898-953ee2 Stopping container test-curl-8898-953ee2 37m Normal Killing pod/test-curl-8898-2de7aa Stopping container test-curl-8898-2de7aa 24m Normal Started pod/test-curl-8889-73b17f Container started 24m Normal Created pod/test-curl-8889-73b17f Container created 24m Normal Pulled pod/test-curl-8889-73b17f Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 664ms (664ms including waiting). Image size: 106100986 bytes. 24m Normal Pulling pod/test-curl-8889-73b17f Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 24m Normal AddedInterface pod/test-curl-8889-73b17f Add eth0 [10.133.0.183/23] from ovn-kubernetes 24m Normal Killing pod/test-curl-8889-73b17f Stopping container test-curl-8889-73b17f 23m Normal AddedInterface pod/test-curl-8883-8c5452 Add eth0 [10.133.0.190/23] from ovn-kubernetes 23m Normal Started pod/test-curl-8883-8c5452 Container started 23m Normal Pulled pod/test-curl-8883-8c5452 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 23m Normal Created pod/test-curl-8883-8c5452 Container created 23m Normal Started pod/test-curl-8883-fbaa72 Container started 23m Normal AddedInterface pod/test-curl-8883-fbaa72 Add eth0 [10.133.0.191/23] from ovn-kubernetes 23m Normal Created pod/test-curl-8883-fbaa72 Container created 23m Normal Pulled pod/test-curl-8883-fbaa72 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 23m Normal Killing pod/test-curl-8883-8c5452 Stopping container test-curl-8883-8c5452 23m Normal Killing pod/test-curl-8883-fbaa72 Stopping container test-curl-8883-fbaa72 14m Normal SuccessfulDelete replicaset/maas-controller-78d6f77c9d Deleted pod: maas-controller-78d6f77c9d-jn4xq 14m Normal Killing pod/maas-controller-78d6f77c9d-jn4xq Stopping container manager 14m Normal SuccessfulCreate replicaset/maas-controller-78d6f77c9d Created pod: maas-controller-78d6f77c9d-b79zr 14m Normal Pulling pod/maas-controller-78d6f77c9d-b79zr Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 14m Normal AddedInterface pod/maas-controller-78d6f77c9d-b79zr Add eth0 [10.132.1.71/23] from ovn-kubernetes 14m Normal Started pod/maas-controller-78d6f77c9d-b79zr Container started 14m Normal Created pod/maas-controller-78d6f77c9d-b79zr Container created 14m Normal Pulled pod/maas-controller-78d6f77c9d-b79zr Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 387ms (387ms including waiting). Image size: 199691795 bytes. 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78d6f77c9d-b79zr_051cefde-cf1e-4b33-af2d-35c07b34de48 became leader 13m Normal Killing pod/maas-controller-78d6f77c9d-b79zr Stopping container manager 13m Normal SuccessfulDelete replicaset/maas-controller-78d6f77c9d Deleted pod: maas-controller-78d6f77c9d-b79zr 13m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-78d6f77c9d from 1 to 0 13m Normal Pulling pod/maas-controller-78d6f77c9d-7kkl4 Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal SuccessfulCreate replicaset/maas-controller-78d6f77c9d Created pod: maas-controller-78d6f77c9d-7kkl4 13m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-78d6f77c9d from 0 to 1 13m Normal AddedInterface pod/maas-controller-78d6f77c9d-7kkl4 Add eth0 [10.132.1.75/23] from ovn-kubernetes 13m Normal Pulled pod/maas-controller-78d6f77c9d-7kkl4 Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 356ms (356ms including waiting). Image size: 199691795 bytes. 13m Normal Created pod/maas-controller-78d6f77c9d-7kkl4 Container created 13m Normal Started pod/maas-controller-78d6f77c9d-7kkl4 Container started 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78d6f77c9d-7kkl4_abf7ae2c-a7ad-4f4b-a239-464f847a6f2f became leader