=== events in opendatahub (2026-10-29T23:19:26Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 56m Normal Scheduled pod/maas-controller-577f596567-895nw Successfully assigned opendatahub/maas-controller-577f596567-895nw to ip-10-0-143-15.ec2.internal 13m Normal Scheduled pod/maas-controller-6c5b76ff8b-rmfgt Successfully assigned opendatahub/maas-controller-6c5b76ff8b-rmfgt to ip-10-0-138-2.ec2.internal 55m Normal Scheduled pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Successfully assigned opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 to ip-10-0-143-15.ec2.internal 41m Normal Scheduled pod/test-curl-9098-b14728 Successfully assigned opendatahub/test-curl-9098-b14728 to ip-10-0-138-2.ec2.internal 39m Normal Scheduled pod/test-curl-9098-820d83 Successfully assigned opendatahub/test-curl-9098-820d83 to ip-10-0-138-2.ec2.internal 25m Normal Scheduled pod/test-curl-9095-a86b40 Successfully assigned opendatahub/test-curl-9095-a86b40 to ip-10-0-143-15.ec2.internal 46m Normal Scheduled pod/test-curl-9095-133e70 Successfully assigned opendatahub/test-curl-9095-133e70 to ip-10-0-128-31.ec2.internal 42m Normal Scheduled pod/test-curl-9092-ed962b Successfully assigned opendatahub/test-curl-9092-ed962b to ip-10-0-138-2.ec2.internal 26m Normal Scheduled pod/test-curl-9092-d4316c Successfully assigned opendatahub/test-curl-9092-d4316c to ip-10-0-143-15.ec2.internal 46m Normal Scheduled pod/test-curl-9092-a20c96 Successfully assigned opendatahub/test-curl-9092-a20c96 to ip-10-0-143-15.ec2.internal 53m Normal Scheduled pod/ai-gateway-controller-5955b644bc-t5bnn Successfully assigned opendatahub/ai-gateway-controller-5955b644bc-t5bnn to ip-10-0-128-31.ec2.internal 26m Normal Scheduled pod/test-curl-9092-72e0f7 Successfully assigned opendatahub/test-curl-9092-72e0f7 to ip-10-0-143-15.ec2.internal 42m Normal Scheduled pod/test-curl-9092-154299 Successfully assigned opendatahub/test-curl-9092-154299 to ip-10-0-138-2.ec2.internal 60m Normal Scheduled pod/odh-observability-584f99c788-nzvk9 Successfully assigned opendatahub/odh-observability-584f99c788-nzvk9 to ip-10-0-143-15.ec2.internal 43m Normal Scheduled pod/test-curl-9092-09835a Successfully assigned opendatahub/test-curl-9092-09835a to ip-10-0-138-2.ec2.internal 59m Normal Scheduled pod/kserve-controller-manager-776cc66d47-w6xdh Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-w6xdh to ip-10-0-143-15.ec2.internal 55m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-cbglz Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-cbglz to ip-10-0-143-15.ec2.internal 60m Normal Scheduled pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Successfully assigned opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd to ip-10-0-128-31.ec2.internal 13m Normal Scheduled pod/maas-controller-6c5b76ff8b-zrlxg Successfully assigned opendatahub/maas-controller-6c5b76ff8b-zrlxg to ip-10-0-138-2.ec2.internal 59m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-4kfs9 to ip-10-0-143-15.ec2.internal 60m Normal Scheduled pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Successfully assigned opendatahub/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 to ip-10-0-138-2.ec2.internal 60m Normal Scheduled pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Successfully assigned opendatahub/kserve-module-controller-manager-5cfb77bf8c-69gvh to ip-10-0-143-15.ec2.internal 53m Normal Scheduled pod/ai-gateway-controller-55b4c89cbd-6szvd Successfully assigned opendatahub/ai-gateway-controller-55b4c89cbd-6szvd to ip-10-0-138-2.ec2.internal 56m Normal Scheduled pod/maas-controller-6c5b76ff8b-g9wkx Successfully assigned opendatahub/maas-controller-6c5b76ff8b-g9wkx to ip-10-0-143-15.ec2.internal 43m Normal Scheduled pod/test-curl-9092-10c45a Successfully assigned opendatahub/test-curl-9092-10c45a to ip-10-0-138-2.ec2.internal 59m Normal Scheduled pod/odh-model-controller-f87fcf56c-v8zwm Successfully assigned opendatahub/odh-model-controller-f87fcf56c-v8zwm to ip-10-0-143-15.ec2.internal 59m Normal Scheduled pod/model-serving-api-75bd89fdfc-45d6d Successfully assigned opendatahub/model-serving-api-75bd89fdfc-45d6d to ip-10-0-143-15.ec2.internal 60m Normal Scheduled pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Successfully assigned opendatahub/opendatahub-operator-controller-manager-6f7c578db4-dsx95 to ip-10-0-143-15.ec2.internal 60m Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 requirements not yet checked 60m Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 one or more requirements couldn't be found 60m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-6f7c578db4 Created pod: opendatahub-operator-controller-manager-6f7c578db4-f4vnd 60m Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 all requirements found, attempting install 60m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-6f7c578db4 Created pod: opendatahub-operator-controller-manager-6f7c578db4-jdgg4 60m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-6f7c578db4 Created pod: opendatahub-operator-controller-manager-6f7c578db4-dsx95 60m Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-6f7c578db4 from 0 to 3 60m Normal AddedInterface pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Add eth0 [10.132.0.31/23] from ovn-kubernetes 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal AddedInterface pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Add eth0 [10.133.0.22/23] from ovn-kubernetes 60m Normal AddedInterface pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Add eth0 [10.134.0.20/23] from ovn-kubernetes 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 Webhook install failed: conversionWebhook not ready 60m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 waiting for install components to report healthy 60m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 60m Warning AppliedWithWarnings installplan/install-nm4cv 1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.852s (2.852s including waiting). Image size: 273709574 bytes. 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Container started 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.777s (2.777s including waiting). Image size: 273709574 bytes. 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Container created 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Container started 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.783s (2.783s including waiting). Image size: 273709574 bytes. 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Container created 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Container created 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Container started 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 294ms (294ms including waiting). Image size: 273709574 bytes. 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Container created 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 350ms (350ms including waiting). Image size: 273709574 bytes. 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Container created 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-dsx95 Container started 60m Normal Pulling pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 60m Normal Pulled pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 427ms (427ms including waiting). Image size: 273709574 bytes. 60m Normal Created pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Container created 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-f4vnd Container started 60m Normal Started pod/opendatahub-operator-controller-manager-6f7c578db4-jdgg4 Container started 60m Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-6f7c578db4-dsx95_f5ab71f1-272e-4415-a928-09a500f495c3 became leader 60m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 install strategy completed with no errors 60m Normal Requested certificate/odh-observability-webhook-cert Created new CertificateRequest resource "odh-observability-webhook-cert-1" 60m Normal SuccessfulCreate replicaset/odh-observability-584f99c788 Created pod: odh-observability-584f99c788-nzvk9 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal CertificateIssued certificaterequest/odh-observability-webhook-cert-1 Certificate fetched from issuer successfully 60m Normal Issuing certificate/odh-observability-webhook-cert The certificate has been successfully issued 60m Normal cert-manager.io certificaterequest/odh-observability-webhook-cert-1 Certificate request has been approved by cert-manager.io 60m Normal Generated certificate/odh-observability-webhook-cert Stored new private key in temporary Secret resource "odh-observability-webhook-cert-v6zlm" 60m Normal ScalingReplicaSet deployment/odh-observability Scaled up replica set odh-observability-584f99c788 from 0 to 1 60m Normal Issuing certificate/odh-observability-webhook-cert Issuing certificate as Secret does not exist 60m Warning FailedMount pod/odh-observability-584f99c788-nzvk9 MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 60m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 60m Normal Pulling pod/odh-observability-584f99c788-nzvk9 Pulling image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" 60m Normal AddedInterface pod/odh-observability-584f99c788-nzvk9 Add eth0 [10.133.0.24/23] from ovn-kubernetes 60m Normal Pulled pod/odh-observability-584f99c788-nzvk9 Successfully pulled image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" in 2.52s (2.52s including waiting). Image size: 154660917 bytes. 60m Normal Created pod/odh-observability-584f99c788-nzvk9 Container created 60m Normal Started pod/odh-observability-584f99c788-nzvk9 Container started 60m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-5cfb77bf8c Created pod: kserve-module-controller-manager-5cfb77bf8c-69gvh 60m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-5cfb77bf8c from 0 to 1 60m Normal Pulling pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 60m Normal AddedInterface pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Add eth0 [10.133.0.26/23] from ovn-kubernetes 60m Normal Started pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Container started 60m Normal Pulled pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.386s (2.386s including waiting). Image size: 195729000 bytes. 60m Normal Created pod/kserve-module-controller-manager-5cfb77bf8c-69gvh Container created 59m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-5cfb77bf8c-69gvh_89bf6dbd-a976-4809-84b9-8698bbddd93c became leader 59m Normal AddedInterface pod/odh-model-controller-f87fcf56c-v8zwm Add eth0 [10.133.0.30/23] from ovn-kubernetes 59m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 59m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 59m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 59m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-4kfs9 59m Normal Pulling pod/odh-model-controller-f87fcf56c-v8zwm Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 59m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Add eth0 [10.133.0.28/23] from ovn-kubernetes 59m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-w6xdh Add eth0 [10.133.0.27/23] from ovn-kubernetes 59m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-f87fcf56c from 0 to 1 59m Normal SuccessfulCreate replicaset/odh-model-controller-f87fcf56c Created pod: odh-model-controller-f87fcf56c-v8zwm 59m Normal Pulling pod/model-serving-api-75bd89fdfc-45d6d Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 59m Normal AddedInterface pod/model-serving-api-75bd89fdfc-45d6d Add eth0 [10.133.0.29/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-w6xdh 59m Normal Pulling pod/kserve-controller-manager-776cc66d47-w6xdh Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 59m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-45d6d 59m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 59m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Container started 59m Normal Created pod/model-serving-api-75bd89fdfc-45d6d Container created 59m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Container created 59m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.015s (3.015s including waiting). Image size: 235100981 bytes. 59m Normal Pulled pod/model-serving-api-75bd89fdfc-45d6d Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.575s (2.575s including waiting). Image size: 184925761 bytes. 59m Normal Started pod/model-serving-api-75bd89fdfc-45d6d Container started 59m Normal Pulled pod/kserve-controller-manager-776cc66d47-w6xdh Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.042s (3.042s including waiting). Image size: 247676581 bytes. 59m Normal Started pod/kserve-controller-manager-776cc66d47-w6xdh Container started 59m Normal Created pod/kserve-controller-manager-776cc66d47-w6xdh Container created 59m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-f87fcf56c-v8zwm_c37d0679-0ce9-4391-8fda-f670cdeb9815 became leader 59m Normal Created pod/odh-model-controller-f87fcf56c-v8zwm Container created 59m Normal Started pod/odh-model-controller-f87fcf56c-v8zwm Container started 59m Normal Pulled pod/odh-model-controller-f87fcf56c-v8zwm Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.524s (3.524s including waiting). Image size: 236755567 bytes. 59m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-w6xdh_d31bb604-763e-4435-87ea-e0b32f972409 became leader 59m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-4kfs9_81a6a142-7838-4cf4-bb41-fd1c6e7e5268 became leader 56m Normal SuccessfulCreate replicaset/maas-controller-577f596567 Created pod: maas-controller-577f596567-895nw 56m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-577f596567 from 0 to 1 56m Normal SuccessfulCreate replicaset/maas-controller-6c5b76ff8b Created pod: maas-controller-6c5b76ff8b-g9wkx 56m Normal Pulling pod/maas-controller-6c5b76ff8b-g9wkx Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 56m Normal AddedInterface pod/maas-controller-6c5b76ff8b-g9wkx Add eth0 [10.133.0.37/23] from ovn-kubernetes 56m Normal AddedInterface pod/maas-controller-577f596567-895nw Add eth0 [10.133.0.36/23] from ovn-kubernetes 56m Normal Pulling pod/maas-controller-577f596567-895nw Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 55m Normal Pulled pod/maas-controller-577f596567-895nw Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 3.072s (3.072s including waiting). Image size: 199691795 bytes. 55m Normal Started pod/maas-controller-6c5b76ff8b-g9wkx Container started 55m Normal Created pod/maas-controller-6c5b76ff8b-g9wkx Container created 55m Normal Pulled pod/maas-controller-6c5b76ff8b-g9wkx Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.805s (2.805s including waiting). Image size: 199691795 bytes. 55m Normal Started pod/maas-controller-577f596567-895nw Container started 55m Normal Created pod/maas-controller-577f596567-895nw Container created 55m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-577f596567-895nw_f2450cb5-c632-448e-bd27-7067759dca6a became leader 55m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-7d94f55f4b from 0 to 1 55m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-7d94f55f4b Created pod: usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 55m Warning FailedMount pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 55m Normal AddedInterface pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Add eth0 [10.133.0.38/23] from ovn-kubernetes 55m Normal Pulling pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 55m Normal Killing pod/maas-controller-577f596567-895nw Stopping container manager 55m Normal SuccessfulDelete replicaset/maas-controller-577f596567 Deleted pod: maas-controller-577f596567-895nw 55m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-577f596567 from 1 to 0 55m Normal Pulled pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.041s (10.041s including waiting). Image size: 1089502687 bytes. 55m Normal Created pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Container created 55m Normal Started pod/usage-logs-tenancy-proxy-7d94f55f4b-cvxs6 Container started 55m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6c5b76ff8b-g9wkx_88ac2e66-c1f8-46f3-906e-5dc690a638ab became leader 55m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-cbglz 55m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 55m Normal Created pod/llmisvc-controller-manager-6466c554d8-cbglz Container created 55m Normal Started pod/llmisvc-controller-manager-6466c554d8-cbglz Container started 55m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-cbglz Add eth0 [10.133.0.40/23] from ovn-kubernetes 55m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-cbglz Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 55m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-cbglz Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 357ms (357ms including waiting). Image size: 235100981 bytes. 54m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-4kfs9 Stopping container manager 54m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 54m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-4kfs9 54m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-cbglz_9a46ee2d-1862-4719-bf82-57d4099e5d57 became leader 53m Normal SuccessfulCreate replicaset/ai-gateway-controller-55b4c89cbd Created pod: ai-gateway-controller-55b4c89cbd-6szvd 53m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-55b4c89cbd from 0 to 1 53m Normal Pulling pod/ai-gateway-controller-55b4c89cbd-6szvd Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" 53m Normal AddedInterface pod/ai-gateway-controller-55b4c89cbd-6szvd Add eth0 [10.134.0.36/23] from ovn-kubernetes 53m Normal Pulled pod/ai-gateway-controller-55b4c89cbd-6szvd Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" in 2.464s (2.464s including waiting). Image size: 183926182 bytes. 53m Normal Created pod/ai-gateway-controller-55b4c89cbd-6szvd Container created 53m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-55b4c89cbd-6szvd_45476c96-d92b-496b-9553-1a53b00ad358 became leader 53m Normal Started pod/ai-gateway-controller-55b4c89cbd-6szvd Container started 53m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5955b644bc from 0 to 1 53m Normal SuccessfulCreate replicaset/ai-gateway-controller-5955b644bc Created pod: ai-gateway-controller-5955b644bc-t5bnn 53m Normal Pulling pod/ai-gateway-controller-5955b644bc-t5bnn Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" 53m Normal AddedInterface pod/ai-gateway-controller-5955b644bc-t5bnn Add eth0 [10.132.0.53/23] from ovn-kubernetes 53m Normal Created pod/ai-gateway-controller-5955b644bc-t5bnn Container created 53m Normal Pulled pod/ai-gateway-controller-5955b644bc-t5bnn Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" in 2.154s (2.154s including waiting). Image size: 183926182 bytes. 53m Normal Started pod/ai-gateway-controller-5955b644bc-t5bnn Container started 53m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-55b4c89cbd from 1 to 0 53m Normal SuccessfulDelete replicaset/ai-gateway-controller-55b4c89cbd Deleted pod: ai-gateway-controller-55b4c89cbd-6szvd 53m Normal Killing pod/ai-gateway-controller-55b4c89cbd-6szvd Stopping container manager 53m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5955b644bc-t5bnn_39723c8e-0f96-4f5a-9664-547bd57802d9 became leader 46m Normal AddedInterface pod/test-curl-9095-133e70 Add eth0 [10.132.0.136/23] from ovn-kubernetes 46m Normal Pulling pod/test-curl-9092-a20c96 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 46m Normal AddedInterface pod/test-curl-9092-a20c96 Add eth0 [10.133.0.69/23] from ovn-kubernetes 46m Normal Started pod/test-curl-9092-a20c96 Container started 46m Normal Created pod/test-curl-9092-a20c96 Container created 46m Normal Pulled pod/test-curl-9092-a20c96 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 586ms (586ms including waiting). Image size: 106100986 bytes. 46m Normal Pulling pod/test-curl-9095-133e70 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 46m Normal Pulled pod/test-curl-9095-133e70 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.083s (2.083s including waiting). Image size: 106100986 bytes. 46m Normal Created pod/test-curl-9095-133e70 Container created 46m Normal Started pod/test-curl-9095-133e70 Container started 46m Normal Killing pod/test-curl-9092-a20c96 Stopping container test-curl-9092-a20c96 46m Normal Killing pod/test-curl-9095-133e70 Stopping container test-curl-9095-133e70 43m Normal AddedInterface pod/test-curl-9092-09835a Add eth0 [10.134.0.107/23] from ovn-kubernetes 43m Normal Pulling pod/test-curl-9092-09835a Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 43m Normal Created pod/test-curl-9092-09835a Container created 43m Normal Started pod/test-curl-9092-09835a Container started 43m Normal Pulled pod/test-curl-9092-09835a Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.725s (2.725s including waiting). Image size: 106100986 bytes. 43m Normal Created pod/test-curl-9092-10c45a Container created 43m Normal Killing pod/test-curl-9092-09835a Stopping container test-curl-9092-09835a 43m Normal AddedInterface pod/test-curl-9092-10c45a Add eth0 [10.134.0.108/23] from ovn-kubernetes 43m Normal Pulled pod/test-curl-9092-10c45a Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 43m Normal Started pod/test-curl-9092-10c45a Container started 42m Normal Killing pod/test-curl-9092-10c45a Stopping container test-curl-9092-10c45a 42m Normal Pulled pod/test-curl-9092-ed962b Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/test-curl-9092-ed962b Add eth0 [10.134.0.109/23] from ovn-kubernetes 42m Normal Created pod/test-curl-9092-ed962b Container created 42m Normal Started pod/test-curl-9092-ed962b Container started 42m Normal Created pod/test-curl-9092-154299 Container created 42m Normal Pulled pod/test-curl-9092-154299 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/test-curl-9092-154299 Add eth0 [10.134.0.110/23] from ovn-kubernetes 42m Normal Started pod/test-curl-9092-154299 Container started 42m Normal Killing pod/test-curl-9092-ed962b Stopping container test-curl-9092-ed962b 42m Normal Killing pod/test-curl-9092-154299 Stopping container test-curl-9092-154299 41m Normal Started pod/test-curl-9098-b14728 Container started 41m Normal Created pod/test-curl-9098-b14728 Container created 41m Normal Pulled pod/test-curl-9098-b14728 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/test-curl-9098-b14728 Add eth0 [10.134.0.120/23] from ovn-kubernetes 41m Normal Killing pod/test-curl-9098-b14728 Stopping container test-curl-9098-b14728 39m Normal Pulled pod/test-curl-9098-820d83 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/test-curl-9098-820d83 Add eth0 [10.134.0.137/23] from ovn-kubernetes 39m Normal Created pod/test-curl-9098-820d83 Container created 39m Normal Started pod/test-curl-9098-820d83 Container started 39m Normal Killing pod/test-curl-9098-820d83 Stopping container test-curl-9098-820d83 26m Normal Pulled pod/test-curl-9092-d4316c Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 26m Normal Started pod/test-curl-9092-d4316c Container started 26m Normal AddedInterface pod/test-curl-9092-d4316c Add eth0 [10.133.0.125/23] from ovn-kubernetes 26m Normal Created pod/test-curl-9092-d4316c Container created 26m Normal AddedInterface pod/test-curl-9092-72e0f7 Add eth0 [10.133.0.126/23] from ovn-kubernetes 26m Normal Started pod/test-curl-9092-72e0f7 Container started 26m Normal Killing pod/test-curl-9092-d4316c Stopping container test-curl-9092-d4316c 26m Normal Created pod/test-curl-9092-72e0f7 Container created 26m Normal Pulled pod/test-curl-9092-72e0f7 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 26m Normal Killing pod/test-curl-9092-72e0f7 Stopping container test-curl-9092-72e0f7 25m Normal AddedInterface pod/test-curl-9095-a86b40 Add eth0 [10.133.0.135/23] from ovn-kubernetes 25m Normal Pulled pod/test-curl-9095-a86b40 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 25m Normal Created pod/test-curl-9095-a86b40 Container created 25m Normal Started pod/test-curl-9095-a86b40 Container started 25m Normal Killing pod/test-curl-9095-a86b40 Stopping container test-curl-9095-a86b40 13m Normal SuccessfulDelete replicaset/maas-controller-6c5b76ff8b Deleted pod: maas-controller-6c5b76ff8b-g9wkx 13m Normal Killing pod/maas-controller-6c5b76ff8b-g9wkx Stopping container manager 13m Normal SuccessfulCreate replicaset/maas-controller-6c5b76ff8b Created pod: maas-controller-6c5b76ff8b-zrlxg 13m Normal Pulling pod/maas-controller-6c5b76ff8b-zrlxg Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal AddedInterface pod/maas-controller-6c5b76ff8b-zrlxg Add eth0 [10.134.1.38/23] from ovn-kubernetes 13m Normal Created pod/maas-controller-6c5b76ff8b-zrlxg Container created 13m Normal Pulled pod/maas-controller-6c5b76ff8b-zrlxg Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 1.815s (1.815s including waiting). Image size: 199691795 bytes. 13m Normal Started pod/maas-controller-6c5b76ff8b-zrlxg Container started 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6c5b76ff8b-zrlxg_30a38771-304b-40c0-b616-e58cc38e2188 became leader 13m Normal SuccessfulDelete replicaset/maas-controller-6c5b76ff8b Deleted pod: maas-controller-6c5b76ff8b-zrlxg 13m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-6c5b76ff8b from 1 to 0 13m Normal Killing pod/maas-controller-6c5b76ff8b-zrlxg Stopping container manager 13m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-6c5b76ff8b from 0 to 1 13m Normal SuccessfulCreate replicaset/maas-controller-6c5b76ff8b Created pod: maas-controller-6c5b76ff8b-rmfgt 13m Normal Pulling pod/maas-controller-6c5b76ff8b-rmfgt Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal AddedInterface pod/maas-controller-6c5b76ff8b-rmfgt Add eth0 [10.134.1.41/23] from ovn-kubernetes 13m Normal Started pod/maas-controller-6c5b76ff8b-rmfgt Container started 13m Normal Pulled pod/maas-controller-6c5b76ff8b-rmfgt Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 439ms (439ms including waiting). Image size: 199691795 bytes. 13m Normal Created pod/maas-controller-6c5b76ff8b-rmfgt Container created 12m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6c5b76ff8b-rmfgt_9487c528-7505-4675-9912-6f856ac04070 became leader