=== events in opendatahub (2026-10-31T08:13:22Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 37m Normal Scheduled pod/maas-controller-5f6b9497b-x9cqc Successfully assigned opendatahub/maas-controller-5f6b9497b-x9cqc to ip-10-0-136-217.ec2.internal 35m Normal Scheduled pod/ai-gateway-controller-7d6c696b49-ss7vq Successfully assigned opendatahub/ai-gateway-controller-7d6c696b49-ss7vq to ip-10-0-140-236.ec2.internal 35m Normal Scheduled pod/ai-gateway-controller-66fd44d76c-d9rzk Successfully assigned opendatahub/ai-gateway-controller-66fd44d76c-d9rzk to ip-10-0-140-236.ec2.internal 38m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-rw52j Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-rw52j to ip-10-0-128-20.ec2.internal 26m Normal Scheduled pod/maas-controller-5f6b9497b-5ln9b Successfully assigned opendatahub/maas-controller-5f6b9497b-5ln9b to ip-10-0-136-217.ec2.internal 26m Normal Scheduled pod/maas-controller-5f6b9497b-fk726 Successfully assigned opendatahub/maas-controller-5f6b9497b-fk726 to ip-10-0-136-217.ec2.internal 37m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h to ip-10-0-136-217.ec2.internal 38m Normal Scheduled pod/kserve-controller-manager-776cc66d47-7vcs5 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-7vcs5 to ip-10-0-128-20.ec2.internal 38m Normal Scheduled pod/model-serving-api-75bd89fdfc-khttk Successfully assigned opendatahub/model-serving-api-75bd89fdfc-khttk to ip-10-0-128-20.ec2.internal 39m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-7jwkr to ip-10-0-136-217.ec2.internal 36m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-82fjs Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-82fjs to ip-10-0-128-20.ec2.internal 38m Normal Scheduled pod/odh-model-controller-769875db56-cp4r2 Successfully assigned opendatahub/odh-model-controller-769875db56-cp4r2 to ip-10-0-128-20.ec2.internal 37m Normal Scheduled pod/maas-controller-579cf7c6d-jkrhs Successfully assigned opendatahub/maas-controller-579cf7c6d-jkrhs to ip-10-0-128-20.ec2.internal 39m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 39m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-7jwkr 39m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Add eth0 [10.133.0.23/23] from ovn-kubernetes 39m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 39m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Container started 39m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Container created 39m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-7jwkr Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.654s (1.654s including waiting). Image size: 195729000 bytes. 38m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-7jwkr_7b2f2f71-3bf0-4921-8405-3ed32494ce3b became leader 38m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-7vcs5 38m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 38m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 38m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-rw52j 38m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 38m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-cp4r2 38m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 38m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-khttk 38m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-rw52j Add eth0 [10.132.0.51/23] from ovn-kubernetes 38m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-7vcs5 Add eth0 [10.132.0.50/23] from ovn-kubernetes 38m Normal Pulling pod/kserve-controller-manager-776cc66d47-7vcs5 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 38m Normal AddedInterface pod/model-serving-api-75bd89fdfc-khttk Add eth0 [10.132.0.52/23] from ovn-kubernetes 38m Normal AddedInterface pod/odh-model-controller-769875db56-cp4r2 Add eth0 [10.132.0.53/23] from ovn-kubernetes 38m Normal Pulling pod/odh-model-controller-769875db56-cp4r2 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 38m Normal Pulling pod/model-serving-api-75bd89fdfc-khttk Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 38m Normal Pulled pod/model-serving-api-75bd89fdfc-khttk Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 3.24s (3.24s including waiting). Image size: 184925761 bytes. 38m Normal Started pod/model-serving-api-75bd89fdfc-khttk Container started 38m Normal Created pod/model-serving-api-75bd89fdfc-khttk Container created 38m Normal Created pod/odh-model-controller-769875db56-cp4r2 Container created 38m Normal Started pod/kserve-controller-manager-776cc66d47-7vcs5 Container started 38m Normal Pulled pod/odh-model-controller-769875db56-cp4r2 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 4.183s (4.183s including waiting). Image size: 236755567 bytes. 38m Normal Pulled pod/kserve-controller-manager-776cc66d47-7vcs5 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 4.23s (4.23s including waiting). Image size: 247676581 bytes. 38m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-rw52j Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 4.358s (4.358s including waiting). Image size: 235100981 bytes. 38m Normal Created pod/kserve-controller-manager-776cc66d47-7vcs5 Container created 38m Normal Started pod/odh-model-controller-769875db56-cp4r2 Container started 38m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-cp4r2_b63ee11b-030a-45a7-ab60-5cbc6c581238 became leader 38m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-7vcs5_f8b8a635-4b4e-4b56-94f5-47e08e2420d7 became leader 37m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-rw52j Container started 37m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-rw52j Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 37m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-rw52j Container created 37m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-rw52j Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 277ms (277ms including waiting). Image size: 235100981 bytes. 37m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-rw52j_c6ecfe2c-cf72-404f-8e2a-3c1b0ace36c6 became leader 37m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-jkrhs 37m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 37m Normal AddedInterface pod/maas-controller-5f6b9497b-x9cqc Add eth0 [10.133.0.25/23] from ovn-kubernetes 37m Normal SuccessfulCreate replicaset/maas-controller-5f6b9497b Created pod: maas-controller-5f6b9497b-x9cqc 37m Normal Pulling pod/maas-controller-5f6b9497b-x9cqc Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 37m Normal Pulling pod/maas-controller-579cf7c6d-jkrhs Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 37m Normal AddedInterface pod/maas-controller-579cf7c6d-jkrhs Add eth0 [10.132.0.54/23] from ovn-kubernetes 37m Normal Pulled pod/maas-controller-579cf7c6d-jkrhs Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.459s (2.459s including waiting). Image size: 200053271 bytes. 37m Normal Started pod/maas-controller-5f6b9497b-x9cqc Container started 37m Normal Created pod/maas-controller-5f6b9497b-x9cqc Container created 37m Normal Pulled pod/maas-controller-5f6b9497b-x9cqc Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.206s (2.206s including waiting). Image size: 200053271 bytes. 37m Normal Started pod/maas-controller-579cf7c6d-jkrhs Container started 37m Normal Created pod/maas-controller-579cf7c6d-jkrhs Container created 37m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-jkrhs_8dad2847-b460-4223-a315-a3f136265099 became leader 37m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-mqt5h 37m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 37m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 37m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Add eth0 [10.133.0.26/23] from ovn-kubernetes 37m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 37m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-jkrhs 37m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 37m Normal Killing pod/maas-controller-579cf7c6d-jkrhs Stopping container manager 37m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Container started 37m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Container created 37m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-mqt5h Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.281s (10.281s including waiting). Image size: 1089502687 bytes. 37m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5f6b9497b-x9cqc_aae3e0ec-67f4-4b6f-8456-5ef379784837 became leader 36m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 36m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-82fjs 36m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-82fjs Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 36m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-82fjs Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 391ms (391ms including waiting). Image size: 235100981 bytes. 36m Normal Started pod/llmisvc-controller-manager-6466c554d8-82fjs Container started 36m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-82fjs Add eth0 [10.132.0.59/23] from ovn-kubernetes 36m Normal Created pod/llmisvc-controller-manager-6466c554d8-82fjs Container created 36m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-rw52j Stopping container manager 36m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 36m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-rw52j 36m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-82fjs_b3b9cbb1-ea21-427d-901d-d59e08c290bb became leader 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7d6c696b49 from 0 to 1 35m Normal SuccessfulCreate replicaset/ai-gateway-controller-7d6c696b49 Created pod: ai-gateway-controller-7d6c696b49-ss7vq 35m Normal Pulling pod/ai-gateway-controller-7d6c696b49-ss7vq Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146" 35m Normal AddedInterface pod/ai-gateway-controller-7d6c696b49-ss7vq Add eth0 [10.134.0.34/23] from ovn-kubernetes 35m Normal Pulled pod/ai-gateway-controller-7d6c696b49-ss7vq Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146" in 2.368s (2.368s including waiting). Image size: 183979436 bytes. 35m Normal Started pod/ai-gateway-controller-7d6c696b49-ss7vq Container started 35m Normal Created pod/ai-gateway-controller-7d6c696b49-ss7vq Container created 35m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7d6c696b49-ss7vq_75f0c814-cc8b-4ab6-b30b-43144ed1d2a4 became leader 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-66fd44d76c from 0 to 1 35m Normal SuccessfulCreate replicaset/ai-gateway-controller-66fd44d76c Created pod: ai-gateway-controller-66fd44d76c-d9rzk 35m Normal Started pod/ai-gateway-controller-66fd44d76c-d9rzk Container started 35m Normal Created pod/ai-gateway-controller-66fd44d76c-d9rzk Container created 35m Normal Pulling pod/ai-gateway-controller-66fd44d76c-d9rzk Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146" 35m Normal Pulled pod/ai-gateway-controller-66fd44d76c-d9rzk Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146" in 284ms (284ms including waiting). Image size: 183979436 bytes. 35m Normal AddedInterface pod/ai-gateway-controller-66fd44d76c-d9rzk Add eth0 [10.134.0.38/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-7d6c696b49 from 1 to 0 35m Normal SuccessfulDelete replicaset/ai-gateway-controller-7d6c696b49 Deleted pod: ai-gateway-controller-7d6c696b49-ss7vq 35m Normal Killing pod/ai-gateway-controller-7d6c696b49-ss7vq Stopping container manager 34m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-66fd44d76c-d9rzk_ebe20a1e-f353-49a8-8346-bcaf1247a7c9 became leader 26m Normal SuccessfulDelete replicaset/maas-controller-5f6b9497b Deleted pod: maas-controller-5f6b9497b-x9cqc 26m Normal Killing pod/maas-controller-5f6b9497b-x9cqc Stopping container manager 26m Normal Pulling pod/maas-controller-5f6b9497b-fk726 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 26m Normal AddedInterface pod/maas-controller-5f6b9497b-fk726 Add eth0 [10.133.0.74/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/maas-controller-5f6b9497b Created pod: maas-controller-5f6b9497b-fk726 26m Normal Created pod/maas-controller-5f6b9497b-fk726 Container created 26m Normal Pulled pod/maas-controller-5f6b9497b-fk726 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 535ms (535ms including waiting). Image size: 200053271 bytes. 26m Normal Started pod/maas-controller-5f6b9497b-fk726 Container started 26m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5f6b9497b-fk726_d0cb66b0-8381-426a-bb97-573dfdf33cd1 became leader 26m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-5f6b9497b from 1 to 0 26m Normal SuccessfulDelete replicaset/maas-controller-5f6b9497b Deleted pod: maas-controller-5f6b9497b-fk726 26m Normal Killing pod/maas-controller-5f6b9497b-fk726 Stopping container manager 26m Normal SuccessfulCreate replicaset/maas-controller-5f6b9497b Created pod: maas-controller-5f6b9497b-5ln9b 26m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-5f6b9497b from 0 to 1 26m Normal Pulling pod/maas-controller-5f6b9497b-5ln9b Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 26m Normal AddedInterface pod/maas-controller-5f6b9497b-5ln9b Add eth0 [10.133.0.75/23] from ovn-kubernetes 26m Normal Started pod/maas-controller-5f6b9497b-5ln9b Container started 26m Normal Created pod/maas-controller-5f6b9497b-5ln9b Container created 26m Normal Pulled pod/maas-controller-5f6b9497b-5ln9b Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 484ms (484ms including waiting). Image size: 200053271 bytes. 25m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5f6b9497b-5ln9b_cae49492-13e5-45df-8918-4a884b9a025a became leader