=== events in opendatahub (2026-11-03T06:30:17Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 29m Normal Scheduled pod/maas-controller-579cf7c6d-tgpjb Successfully assigned opendatahub/maas-controller-579cf7c6d-tgpjb to ip-10-0-138-242.ec2.internal 29m Normal Scheduled pod/kserve-controller-manager-776cc66d47-bm44l Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-bm44l to ip-10-0-138-242.ec2.internal 17m Normal Scheduled pod/maas-controller-678d788c65-zrb8q Successfully assigned opendatahub/maas-controller-678d788c65-zrb8q to ip-10-0-141-67.ec2.internal 28m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-pkcvl Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-pkcvl to ip-10-0-138-242.ec2.internal 30m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-ntcx4 to ip-10-0-138-242.ec2.internal 26m Normal Scheduled pod/ai-gateway-controller-7f7fb74d9c-c8g4b Successfully assigned opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b to ip-10-0-132-233.ec2.internal 18m Normal Scheduled pod/maas-controller-678d788c65-5rsst Successfully assigned opendatahub/maas-controller-678d788c65-5rsst to ip-10-0-141-67.ec2.internal 27m Normal Scheduled pod/ai-gateway-controller-b59d66cb5-qbstw Successfully assigned opendatahub/ai-gateway-controller-b59d66cb5-qbstw to ip-10-0-132-233.ec2.internal 29m Normal Scheduled pod/maas-controller-678d788c65-2vlhl Successfully assigned opendatahub/maas-controller-678d788c65-2vlhl to ip-10-0-141-67.ec2.internal 29m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-wjj7p to ip-10-0-138-242.ec2.internal 29m Normal Scheduled pod/model-serving-api-75bd89fdfc-9q8nl Successfully assigned opendatahub/model-serving-api-75bd89fdfc-9q8nl to ip-10-0-138-242.ec2.internal 29m Normal Scheduled pod/odh-model-controller-769875db56-kcp47 Successfully assigned opendatahub/odh-model-controller-769875db56-kcp47 to ip-10-0-138-242.ec2.internal 29m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gth2s to ip-10-0-141-67.ec2.internal 30m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 30m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-ntcx4 30m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Add eth0 [10.134.0.27/23] from ovn-kubernetes 30m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 30m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Container started 30m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.513s (1.513s including waiting). Image size: 195729000 bytes. 30m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-ntcx4 Container created 30m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-ntcx4_892d1d1d-6ac4-4aad-925f-56ae17e00a70 became leader 29m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 29m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-9q8nl 29m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-wjj7p 29m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-kcp47 29m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 29m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 29m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 29m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-bm44l 29m Normal Pulling pod/odh-model-controller-769875db56-kcp47 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 29m Normal Pulling pod/kserve-controller-manager-776cc66d47-bm44l Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 29m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-bm44l Add eth0 [10.134.0.32/23] from ovn-kubernetes 29m Normal AddedInterface pod/model-serving-api-75bd89fdfc-9q8nl Add eth0 [10.134.0.34/23] from ovn-kubernetes 29m Normal Pulling pod/model-serving-api-75bd89fdfc-9q8nl Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 29m Normal AddedInterface pod/odh-model-controller-769875db56-kcp47 Add eth0 [10.134.0.35/23] from ovn-kubernetes 29m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Add eth0 [10.134.0.33/23] from ovn-kubernetes 29m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 29m Normal Started pod/model-serving-api-75bd89fdfc-9q8nl Container started 29m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Container created 29m Normal Started pod/kserve-controller-manager-776cc66d47-bm44l Container started 29m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Container started 29m Normal Created pod/kserve-controller-manager-776cc66d47-bm44l Container created 29m Normal Pulled pod/kserve-controller-manager-776cc66d47-bm44l Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.13s (3.13s including waiting). Image size: 247676581 bytes. 29m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.119s (3.119s including waiting). Image size: 235100981 bytes. 29m Normal Pulled pod/odh-model-controller-769875db56-kcp47 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.635s (3.635s including waiting). Image size: 236755567 bytes. 29m Normal Created pod/model-serving-api-75bd89fdfc-9q8nl Container created 29m Normal Pulled pod/model-serving-api-75bd89fdfc-9q8nl Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.734s (2.734s including waiting). Image size: 184925761 bytes. 29m Normal Created pod/odh-model-controller-769875db56-kcp47 Container created 29m Normal Started pod/odh-model-controller-769875db56-kcp47 Container started 29m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-kcp47_ac4bd5ad-695d-44d6-9722-b454f330c3e6 became leader 29m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-bm44l_dfabc2b9-a42d-483a-9ef9-d9d2af2baef4 became leader 29m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-wjj7p_57c3e004-75b1-412e-9947-6f95c9edef14 became leader 29m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-tgpjb 29m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 29m Normal SuccessfulCreate replicaset/maas-controller-678d788c65 Created pod: maas-controller-678d788c65-2vlhl 29m Normal AddedInterface pod/maas-controller-678d788c65-2vlhl Add eth0 [10.133.0.29/23] from ovn-kubernetes 29m Normal Pulling pod/maas-controller-678d788c65-2vlhl Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 29m Normal Pulling pod/maas-controller-579cf7c6d-tgpjb Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 29m Normal AddedInterface pod/maas-controller-579cf7c6d-tgpjb Add eth0 [10.134.0.36/23] from ovn-kubernetes 29m Normal Pulled pod/maas-controller-579cf7c6d-tgpjb Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.461s (2.461s including waiting). Image size: 200053271 bytes. 29m Normal Started pod/maas-controller-579cf7c6d-tgpjb Container started 29m Normal Started pod/maas-controller-678d788c65-2vlhl Container started 29m Normal Created pod/maas-controller-678d788c65-2vlhl Container created 29m Normal Pulled pod/maas-controller-678d788c65-2vlhl Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.22s (2.22s including waiting). Image size: 200053271 bytes. 29m Normal Created pod/maas-controller-579cf7c6d-tgpjb Container created 29m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-tgpjb_1816dab6-5d40-4f5a-9fd6-6e0c0af57439 became leader 29m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 29m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-gth2s 29m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Add eth0 [10.133.0.30/23] from ovn-kubernetes 29m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 29m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-tgpjb 29m Normal Killing pod/maas-controller-579cf7c6d-tgpjb Stopping container manager 29m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 29m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 12.567s (12.567s including waiting). Image size: 1089502687 bytes. 29m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Container created 29m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-gth2s Container started 28m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-678d788c65-2vlhl_977b6963-5b86-4828-a064-a58aa0e34dcc became leader 28m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 28m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-pkcvl 28m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-pkcvl Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 28m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-pkcvl Add eth0 [10.134.0.41/23] from ovn-kubernetes 28m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-pkcvl Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 377ms (377ms including waiting). Image size: 235100981 bytes. 28m Normal Created pod/llmisvc-controller-manager-6466c554d8-pkcvl Container created 28m Normal Started pod/llmisvc-controller-manager-6466c554d8-pkcvl Container started 27m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 27m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-wjj7p 27m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-wjj7p Stopping container manager 27m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-pkcvl_2c9c03dc-1c79-4239-8ed7-b718cb73bff9 became leader 27m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-b59d66cb5 from 0 to 1 27m Warning FailedMount pod/ai-gateway-controller-b59d66cb5-qbstw MountVolume.SetUp failed for volume "metrics-certs" : secret "ai-gateway-controller-metrics-tls" not found 27m Normal SuccessfulCreate replicaset/ai-gateway-controller-b59d66cb5 Created pod: ai-gateway-controller-b59d66cb5-qbstw 27m Normal Pulling pod/ai-gateway-controller-b59d66cb5-qbstw Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" 27m Normal AddedInterface pod/ai-gateway-controller-b59d66cb5-qbstw Add eth0 [10.132.0.48/23] from ovn-kubernetes 27m Normal Created pod/ai-gateway-controller-b59d66cb5-qbstw Container created 27m Normal Started pod/ai-gateway-controller-b59d66cb5-qbstw Container started 27m Normal Pulled pod/ai-gateway-controller-b59d66cb5-qbstw Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" in 3.185s (3.185s including waiting). Image size: 183983521 bytes. 27m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-b59d66cb5-qbstw_32c96c8f-e15c-486d-8696-c15c884ce119 became leader 26m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7f7fb74d9c from 0 to 1 26m Normal Pulling pod/ai-gateway-controller-7f7fb74d9c-c8g4b Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" 26m Normal SuccessfulCreate replicaset/ai-gateway-controller-7f7fb74d9c Created pod: ai-gateway-controller-7f7fb74d9c-c8g4b 26m Normal AddedInterface pod/ai-gateway-controller-7f7fb74d9c-c8g4b Add eth0 [10.132.0.52/23] from ovn-kubernetes 26m Normal Created pod/ai-gateway-controller-7f7fb74d9c-c8g4b Container created 26m Normal Started pod/ai-gateway-controller-7f7fb74d9c-c8g4b Container started 26m Normal Pulled pod/ai-gateway-controller-7f7fb74d9c-c8g4b Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" in 390ms (390ms including waiting). Image size: 183983521 bytes. 26m Normal Killing pod/ai-gateway-controller-b59d66cb5-qbstw Stopping container manager 26m Normal SuccessfulDelete replicaset/ai-gateway-controller-b59d66cb5 Deleted pod: ai-gateway-controller-b59d66cb5-qbstw 26m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-b59d66cb5 from 1 to 0 26m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7f7fb74d9c-c8g4b_cb38e0fb-505e-41b3-9cd9-47d48b89f263 became leader 18m Normal Killing pod/maas-controller-678d788c65-2vlhl Stopping container manager 18m Normal SuccessfulDelete replicaset/maas-controller-678d788c65 Deleted pod: maas-controller-678d788c65-2vlhl 18m Normal SuccessfulCreate replicaset/maas-controller-678d788c65 Created pod: maas-controller-678d788c65-5rsst 18m Normal Pulling pod/maas-controller-678d788c65-5rsst Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal AddedInterface pod/maas-controller-678d788c65-5rsst Add eth0 [10.133.0.79/23] from ovn-kubernetes 18m Normal Pulled pod/maas-controller-678d788c65-5rsst Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 416ms (416ms including waiting). Image size: 200053271 bytes. 18m Normal Created pod/maas-controller-678d788c65-5rsst Container created 18m Normal Started pod/maas-controller-678d788c65-5rsst Container started 18m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-678d788c65-5rsst_ec9ea667-4d04-4e1f-88ff-778daef0ebac became leader 17m Normal SuccessfulDelete replicaset/maas-controller-678d788c65 Deleted pod: maas-controller-678d788c65-5rsst 17m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-678d788c65 from 1 to 0 17m Normal Killing pod/maas-controller-678d788c65-5rsst Stopping container manager 17m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-678d788c65 from 0 to 1 17m Normal SuccessfulCreate replicaset/maas-controller-678d788c65 Created pod: maas-controller-678d788c65-zrb8q 17m Normal Started pod/maas-controller-678d788c65-zrb8q Container started 17m Normal AddedInterface pod/maas-controller-678d788c65-zrb8q Add eth0 [10.133.0.80/23] from ovn-kubernetes 17m Normal Pulling pod/maas-controller-678d788c65-zrb8q Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 17m Normal Pulled pod/maas-controller-678d788c65-zrb8q Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 315ms (315ms including waiting). Image size: 200053271 bytes. 17m Normal Created pod/maas-controller-678d788c65-zrb8q Container created 17m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-678d788c65-zrb8q_9a32380c-41e2-4554-89d6-9c43bd3cc365 became leader