=== events in opendatahub (2026-11-04T06:12:27Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 4m56s Normal Scheduled pod/test-curl-7281-41f7f7 Successfully assigned opendatahub/test-curl-7281-41f7f7 to ip-10-0-134-45.us-west-1.compute.internal 15m Normal Scheduled pod/maas-controller-6684d95577-2m9xk Successfully assigned opendatahub/maas-controller-6684d95577-2m9xk to ip-10-0-134-45.us-west-1.compute.internal 13m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-ddvq8 Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-ddvq8 to ip-10-0-134-45.us-west-1.compute.internal 15m Normal Scheduled pod/maas-controller-8b95c9c66-vm7gw Successfully assigned opendatahub/maas-controller-8b95c9c66-vm7gw to ip-10-0-134-45.us-west-1.compute.internal 14m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-cz48n Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-cz48n to ip-10-0-134-45.us-west-1.compute.internal 15m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-gmr6l to ip-10-0-133-0.us-west-1.compute.internal 14m Normal Scheduled pod/kserve-controller-manager-776cc66d47-5sp6z Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-5sp6z to ip-10-0-134-45.us-west-1.compute.internal 14m Normal Scheduled pod/model-serving-api-75bd89fdfc-jvh5s Successfully assigned opendatahub/model-serving-api-75bd89fdfc-jvh5s to ip-10-0-134-45.us-west-1.compute.internal 11m Normal Scheduled pod/ai-gateway-controller-7958bf97d9-b55f4 Successfully assigned opendatahub/ai-gateway-controller-7958bf97d9-b55f4 to ip-10-0-133-0.us-west-1.compute.internal 11m Normal Scheduled pod/ai-gateway-controller-c4d85ff99-nc79m Successfully assigned opendatahub/ai-gateway-controller-c4d85ff99-nc79m to ip-10-0-133-0.us-west-1.compute.internal 14m Normal Scheduled pod/odh-model-controller-769875db56-j2scg Successfully assigned opendatahub/odh-model-controller-769875db56-j2scg to ip-10-0-134-45.us-west-1.compute.internal 15m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh to ip-10-0-134-45.us-west-1.compute.internal 4m56s Normal Scheduled pod/test-curl-7290-1e1f2a Successfully assigned opendatahub/test-curl-7290-1e1f2a to ip-10-0-143-195.us-west-1.compute.internal 15m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 15m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Add eth0 [10.132.0.49/23] from ovn-kubernetes 15m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 15m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-gmr6l 15m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Container started 15m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Container created 15m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-gmr6l Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.696s (2.696s including waiting). Image size: 195729000 bytes. 15m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-gmr6l_27c92ccb-bd79-4fe5-a267-ff7703526545 became leader 15m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-8b95c9c66 from 0 to 1 15m Warning FailedMount pod/maas-controller-8b95c9c66-vm7gw MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 15m Normal SuccessfulCreate replicaset/maas-controller-6684d95577 Created pod: maas-controller-6684d95577-2m9xk 15m Normal SuccessfulCreate replicaset/maas-controller-8b95c9c66 Created pod: maas-controller-8b95c9c66-vm7gw 15m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-6684d95577 from 0 to 1 15m Normal Pulling pod/maas-controller-8b95c9c66-vm7gw Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 15m Normal AddedInterface pod/maas-controller-8b95c9c66-vm7gw Add eth0 [10.134.0.30/23] from ovn-kubernetes 15m Normal AddedInterface pod/maas-controller-6684d95577-2m9xk Add eth0 [10.134.0.31/23] from ovn-kubernetes 15m Normal Pulled pod/maas-controller-8b95c9c66-vm7gw Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 3.172s (3.172s including waiting). Image size: 200103442 bytes. 15m Normal Created pod/maas-controller-8b95c9c66-vm7gw Container created 15m Normal Started pod/maas-controller-8b95c9c66-vm7gw Container started 15m Normal Pulled pod/maas-controller-6684d95577-2m9xk Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 3.384s (3.384s including waiting). Image size: 200103442 bytes. 15m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-8b95c9c66-vm7gw_0843d26e-17de-48fa-96c8-e849fc1e5893 became leader 15m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh MountVolume.SetUp failed for volume "proxy-source" : configmap "usage-tenancy-proxy-source" not found 15m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-f9tvh 15m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 15m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 15m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Add eth0 [10.134.0.32/23] from ovn-kubernetes 15m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 15m Normal SuccessfulDelete replicaset/maas-controller-8b95c9c66 Deleted pod: maas-controller-8b95c9c66-vm7gw 15m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-8b95c9c66 from 1 to 0 15m Normal Killing pod/maas-controller-8b95c9c66-vm7gw Stopping container manager 14m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 12.822s (12.822s including waiting). Image size: 1089502687 bytes. 14m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Container started 14m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-f9tvh Container created 14m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6684d95577-2m9xk_d72f58c0-ad96-4dcf-a6ed-44a21194db3e became leader 14m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 14m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-5sp6z 14m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-5sp6z Add eth0 [10.134.0.33/23] from ovn-kubernetes 14m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-cz48n Add eth0 [10.134.0.34/23] from ovn-kubernetes 14m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-cz48n Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 14m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-jvh5s 14m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 14m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-j2scg 14m Normal AddedInterface pod/model-serving-api-75bd89fdfc-jvh5s Add eth0 [10.134.0.35/23] from ovn-kubernetes 14m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-cz48n 14m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 14m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 14m Normal Pulling pod/model-serving-api-75bd89fdfc-jvh5s Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 14m Normal Pulling pod/odh-model-controller-769875db56-j2scg Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 14m Normal AddedInterface pod/odh-model-controller-769875db56-j2scg Add eth0 [10.134.0.36/23] from ovn-kubernetes 14m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-cz48n Container started 14m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-cz48n Container created 14m Normal Pulled pod/kserve-controller-manager-776cc66d47-5sp6z Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.395s (3.395s including waiting). Image size: 247676581 bytes. 14m Normal Created pod/model-serving-api-75bd89fdfc-jvh5s Container created 14m Normal Pulled pod/model-serving-api-75bd89fdfc-jvh5s Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 3.278s (3.278s including waiting). Image size: 184925761 bytes. 14m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-cz48n Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.24s (3.24s including waiting). Image size: 235100981 bytes. 14m Normal Started pod/model-serving-api-75bd89fdfc-jvh5s Container started 14m Normal Created pod/odh-model-controller-769875db56-j2scg Container created 14m Normal Pulled pod/odh-model-controller-769875db56-j2scg Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.87s (3.87s including waiting). Image size: 236755567 bytes. 14m Normal Started pod/odh-model-controller-769875db56-j2scg Container started 14m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-5sp6z_7330a671-40fe-45ae-bc5d-7dfe10a993c8 became leader 14m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-cz48n_11282d12-6289-4226-a2ef-6bfd3618a324 became leader 14m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-j2scg_c9ca02a6-353b-4ed1-8a70-00019bb96190 became leader 13m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 13m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-ddvq8 13m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-ddvq8 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 13m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-ddvq8 Add eth0 [10.134.0.37/23] from ovn-kubernetes 13m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-ddvq8 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 812ms (812ms including waiting). Image size: 235100981 bytes. 13m Normal Created pod/llmisvc-controller-manager-6466c554d8-ddvq8 Container created 13m Normal Started pod/llmisvc-controller-manager-6466c554d8-ddvq8 Container started 12m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-cz48n Stopping container manager 12m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-cz48n 12m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 12m Normal Pulled pod/maas-controller-6684d95577-2m9xk Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 1.121s (1.121s including waiting). Image size: 200103442 bytes. 12m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-ddvq8_fe8b8e2b-42b3-4a52-999f-e8d32cfd8413 became leader 12m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6684d95577-2m9xk_0ec97233-a001-4848-92b8-18d8c07ae445 became leader 11m Normal Pulling pod/ai-gateway-controller-7958bf97d9-b55f4 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 11m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7958bf97d9 from 0 to 1 11m Normal SuccessfulCreate replicaset/ai-gateway-controller-7958bf97d9 Created pod: ai-gateway-controller-7958bf97d9-b55f4 11m Normal AddedInterface pod/ai-gateway-controller-7958bf97d9-b55f4 Add eth0 [10.132.0.55/23] from ovn-kubernetes 11m Normal Started pod/ai-gateway-controller-7958bf97d9-b55f4 Container started 11m Normal Pulled pod/ai-gateway-controller-7958bf97d9-b55f4 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 5.611s (5.611s including waiting). Image size: 184144295 bytes. 11m Normal Created pod/ai-gateway-controller-7958bf97d9-b55f4 Container created 11m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7958bf97d9-b55f4_385ada65-89d8-4ada-bf5d-37fe8a221049 became leader 11m Normal AddedInterface pod/ai-gateway-controller-c4d85ff99-nc79m Add eth0 [10.132.0.58/23] from ovn-kubernetes 11m Normal SuccessfulCreate replicaset/ai-gateway-controller-c4d85ff99 Created pod: ai-gateway-controller-c4d85ff99-nc79m 11m Normal Pulling pod/ai-gateway-controller-c4d85ff99-nc79m Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 11m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-c4d85ff99 from 0 to 1 11m Normal Created pod/ai-gateway-controller-c4d85ff99-nc79m Container created 11m Normal Pulled pod/ai-gateway-controller-c4d85ff99-nc79m Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 958ms (958ms including waiting). Image size: 184144295 bytes. 11m Normal Started pod/ai-gateway-controller-c4d85ff99-nc79m Container started 10m Normal Killing pod/ai-gateway-controller-7958bf97d9-b55f4 Stopping container manager 10m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-7958bf97d9 from 1 to 0 10m Normal SuccessfulDelete replicaset/ai-gateway-controller-7958bf97d9 Deleted pod: ai-gateway-controller-7958bf97d9-b55f4 10m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-c4d85ff99-nc79m_1484c742-6f40-4693-bd1b-6be14a9715dd became leader 4m55s Normal AddedInterface pod/test-curl-7281-41f7f7 Add eth0 [10.134.0.51/23] from ovn-kubernetes 4m55s Normal Pulling pod/test-curl-7281-41f7f7 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 4m55s Normal Pulling pod/test-curl-7290-1e1f2a Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 4m55s Normal AddedInterface pod/test-curl-7290-1e1f2a Add eth0 [10.133.0.55/23] from ovn-kubernetes 4m54s Normal Pulled pod/test-curl-7281-41f7f7 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 1.009s (1.009s including waiting). Image size: 106100986 bytes. 4m54s Normal Started pod/test-curl-7281-41f7f7 Container started 4m54s Normal Created pod/test-curl-7281-41f7f7 Container created 4m53s Normal Pulled pod/test-curl-7290-1e1f2a Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.456s (2.456s including waiting). Image size: 106100986 bytes. 4m52s Normal Created pod/test-curl-7290-1e1f2a Container created 4m52s Normal Started pod/test-curl-7290-1e1f2a Container started 4m52s Normal Killing pod/test-curl-7281-41f7f7 Stopping container test-curl-7281-41f7f7 4m50s Normal Killing pod/test-curl-7290-1e1f2a Stopping container test-curl-7290-1e1f2a 13s Normal Pulling pod/maas-controller-6684d95577-2m9xk Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 12s Normal Pulled pod/maas-controller-6684d95577-2m9xk Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 855ms (855ms including waiting). Image size: 200103442 bytes. 12s Normal Created pod/maas-controller-6684d95577-2m9xk Container created 12s Normal Started pod/maas-controller-6684d95577-2m9xk Container started 11s Normal Pulling pod/kserve-controller-manager-776cc66d47-5sp6z Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 10s Normal Created pod/kserve-controller-manager-776cc66d47-5sp6z Container created 10s Normal Pulled pod/kserve-controller-manager-776cc66d47-5sp6z Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 814ms (814ms including waiting). Image size: 247676581 bytes. 10s Normal Started pod/kserve-controller-manager-776cc66d47-5sp6z Container started 1s Warning Unhealthy pod/maas-controller-6684d95577-2m9xk Readiness probe failed: Get "http://10.134.0.31:8081/readyz": dial tcp 10.134.0.31:8081: connect: connection refused 1s Warning BackOff pod/maas-controller-6684d95577-2m9xk Back-off restarting failed container manager in pod maas-controller-6684d95577-2m9xk_opendatahub(3a430fb7-e8ae-4f64-8f57-e5ea2b808725)