=== events in opendatahub (2026-10-31T18:45:41Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 10m Normal Scheduled pod/maas-controller-5b4695597-2wv54 Successfully assigned opendatahub/maas-controller-5b4695597-2wv54 to ip-10-0-141-104.us-west-2.compute.internal 10m Normal Scheduled pod/kserve-controller-manager-776cc66d47-7vmsc Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-7vmsc to ip-10-0-140-53.us-west-2.compute.internal 7m43s Normal Scheduled pod/ai-gateway-controller-5f85cd9cfc-rk8cr Successfully assigned opendatahub/ai-gateway-controller-5f85cd9cfc-rk8cr to ip-10-0-140-53.us-west-2.compute.internal 9m38s Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-8zbnk Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-8zbnk to ip-10-0-141-104.us-west-2.compute.internal 10m Normal Scheduled pod/maas-controller-579cf7c6d-dckts Successfully assigned opendatahub/maas-controller-579cf7c6d-dckts to ip-10-0-141-104.us-west-2.compute.internal 10m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-scqhd Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-scqhd to ip-10-0-141-104.us-west-2.compute.internal 10m Normal Scheduled pod/odh-model-controller-769875db56-22fb4 Successfully assigned opendatahub/odh-model-controller-769875db56-22fb4 to ip-10-0-140-53.us-west-2.compute.internal 12m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-ch2bw to ip-10-0-141-104.us-west-2.compute.internal 10m Normal Scheduled pod/model-serving-api-75bd89fdfc-mdx5p Successfully assigned opendatahub/model-serving-api-75bd89fdfc-mdx5p to ip-10-0-140-53.us-west-2.compute.internal 7m26s Normal Scheduled pod/ai-gateway-controller-7779c685d8-d2f7f Successfully assigned opendatahub/ai-gateway-controller-7779c685d8-d2f7f to ip-10-0-136-243.us-west-2.compute.internal 10m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs to ip-10-0-141-104.us-west-2.compute.internal 12m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-ch2bw 12m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 11m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 11m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Add eth0 [10.132.0.43/23] from ovn-kubernetes 11m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Container started 11m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Container created 11m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 3.118s (3.118s including waiting). Image size: 195729000 bytes. 11m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-ch2bw_644dbdc9-9a58-4680-95d5-77880d566930 became leader 10m Normal SuccessfulCreate replicaset/maas-controller-5b4695597 Created pod: maas-controller-5b4695597-2wv54 10m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 10m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-5b4695597 from 0 to 1 10m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-dckts 10m Normal AddedInterface pod/maas-controller-579cf7c6d-dckts Add eth0 [10.132.0.44/23] from ovn-kubernetes 10m Normal AddedInterface pod/maas-controller-5b4695597-2wv54 Add eth0 [10.132.0.45/23] from ovn-kubernetes 10m Normal Pulling pod/maas-controller-579cf7c6d-dckts Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 10m Normal Pulling pod/maas-controller-5b4695597-2wv54 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 10m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 10m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-scqhd Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 10m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 10m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-7vmsc 10m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-7vmsc Add eth0 [10.134.0.27/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 10m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-scqhd 10m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 10m Normal AddedInterface pod/model-serving-api-75bd89fdfc-mdx5p Add eth0 [10.134.0.28/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-22fb4 10m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-scqhd Add eth0 [10.132.0.46/23] from ovn-kubernetes 10m Normal Pulling pod/kserve-controller-manager-776cc66d47-7vmsc Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 10m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-mdx5p 10m Normal Pulling pod/odh-model-controller-769875db56-22fb4 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 10m Normal AddedInterface pod/odh-model-controller-769875db56-22fb4 Add eth0 [10.134.0.29/23] from ovn-kubernetes 10m Normal Pulling pod/model-serving-api-75bd89fdfc-mdx5p Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 10m Normal Pulled pod/maas-controller-579cf7c6d-dckts Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.718s (3.718s including waiting). Image size: 200053271 bytes. 10m Normal Started pod/maas-controller-5b4695597-2wv54 Container started 10m Normal Created pod/maas-controller-5b4695597-2wv54 Container created 10m Normal Pulled pod/maas-controller-5b4695597-2wv54 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.537s (3.537s including waiting). Image size: 200053271 bytes. 10m Normal Created pod/maas-controller-579cf7c6d-dckts Container created 10m Normal Started pod/maas-controller-579cf7c6d-dckts Container started 10m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-scqhd Container created 10m Normal Created pod/kserve-controller-manager-776cc66d47-7vmsc Container created 10m Normal Started pod/kserve-controller-manager-776cc66d47-7vmsc Container started 10m Normal Pulled pod/kserve-controller-manager-776cc66d47-7vmsc Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.976s (2.976s including waiting). Image size: 247676581 bytes. 10m Normal Pulled pod/model-serving-api-75bd89fdfc-mdx5p Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.922s (2.922s including waiting). Image size: 184925761 bytes. 10m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-scqhd Container started 10m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-scqhd Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.583s (2.583s including waiting). Image size: 235100981 bytes. 10m Normal Created pod/odh-model-controller-769875db56-22fb4 Container created 10m Normal Pulled pod/odh-model-controller-769875db56-22fb4 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.657s (3.657s including waiting). Image size: 236755567 bytes. 10m Normal Started pod/model-serving-api-75bd89fdfc-mdx5p Container started 10m Normal Created pod/model-serving-api-75bd89fdfc-mdx5p Container created 10m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-7vmsc_29a3a004-2ed7-466c-92c3-c11c954648b8 became leader 10m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5b4695597-2wv54_a77ad593-55fb-411b-b584-291a2475307e became leader 10m Normal Started pod/odh-model-controller-769875db56-22fb4 Container started 10m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-22fb4_73c4f173-dfb6-44c9-8dfd-a79da8d93a0a became leader 10m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 10m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-scqhd_990d66e9-8add-49c0-8724-ac334bc1d9a1 became leader 10m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-8cgzs 10m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 10m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs MountVolume.SetUp failed for volume "proxy-source" : configmap "usage-tenancy-proxy-source" not found 10m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 10m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Add eth0 [10.132.0.47/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 10m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-dckts 10m Normal Killing pod/maas-controller-579cf7c6d-dckts Stopping container manager 10m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Container created 10m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 13.236s (13.236s including waiting). Image size: 1089502687 bytes. 10m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Container started 9m38s Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 9m38s Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-8zbnk 9m38s Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-8zbnk Add eth0 [10.132.0.51/23] from ovn-kubernetes 9m37s Normal Started pod/llmisvc-controller-manager-6466c554d8-8zbnk Container started 9m37s Normal Created pod/llmisvc-controller-manager-6466c554d8-8zbnk Container created 9m37s Normal Pulled pod/llmisvc-controller-manager-6466c554d8-8zbnk Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 753ms (753ms including waiting). Image size: 235100981 bytes. 9m37s Normal Pulling pod/llmisvc-controller-manager-6466c554d8-8zbnk Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 9m5s Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-scqhd 9m5s Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 9m5s Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-scqhd Stopping container manager 8m51s Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-8zbnk_3d6e3c07-500c-4ebe-ad4d-819b5939f1c1 became leader 7m44s Normal SuccessfulCreate replicaset/ai-gateway-controller-5f85cd9cfc Created pod: ai-gateway-controller-5f85cd9cfc-rk8cr 7m44s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5f85cd9cfc from 0 to 1 7m43s Normal AddedInterface pod/ai-gateway-controller-5f85cd9cfc-rk8cr Add eth0 [10.134.0.39/23] from ovn-kubernetes 7m43s Normal Pulling pod/ai-gateway-controller-5f85cd9cfc-rk8cr Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 7m38s Normal Pulled pod/ai-gateway-controller-5f85cd9cfc-rk8cr Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 4.723s (4.723s including waiting). Image size: 183987623 bytes. 7m38s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5f85cd9cfc-rk8cr_a29c767b-9da3-4a60-a801-38404fa0b047 became leader 7m38s Normal Started pod/ai-gateway-controller-5f85cd9cfc-rk8cr Container started 7m38s Normal Created pod/ai-gateway-controller-5f85cd9cfc-rk8cr Container created 7m27s Normal SuccessfulCreate replicaset/ai-gateway-controller-7779c685d8 Created pod: ai-gateway-controller-7779c685d8-d2f7f 7m27s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7779c685d8 from 0 to 1 7m26s Normal AddedInterface pod/ai-gateway-controller-7779c685d8-d2f7f Add eth0 [10.133.0.36/23] from ovn-kubernetes 7m26s Normal Pulling pod/ai-gateway-controller-7779c685d8-d2f7f Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" 7m23s Normal Started pod/ai-gateway-controller-7779c685d8-d2f7f Container started 7m23s Normal Pulled pod/ai-gateway-controller-7779c685d8-d2f7f Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177" in 3.052s (3.052s including waiting). Image size: 183987623 bytes. 7m23s Normal Created pod/ai-gateway-controller-7779c685d8-d2f7f Container created 7m15s Warning NodeNotReady pod/maas-controller-5b4695597-2wv54 Node is not ready 7m15s Warning NodeNotReady pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Node is not ready 7m15s Warning NodeNotReady pod/llmisvc-controller-manager-6466c554d8-8zbnk Node is not ready 7m14s Warning NodeNotReady pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Node is not ready 7m11s Normal SuccessfulDelete replicaset/ai-gateway-controller-5f85cd9cfc Deleted pod: ai-gateway-controller-5f85cd9cfc-rk8cr 7m11s Normal Killing pod/ai-gateway-controller-5f85cd9cfc-rk8cr Stopping container manager 7m11s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5f85cd9cfc from 1 to 0 7m4s Normal TaintManagerEviction pod/llmisvc-controller-manager-6466c554d8-8zbnk Cancelling deletion of Pod opendatahub/llmisvc-controller-manager-6466c554d8-8zbnk 7m4s Normal TaintManagerEviction pod/maas-controller-5b4695597-2wv54 Cancelling deletion of Pod opendatahub/maas-controller-5b4695597-2wv54 7m4s Normal TaintManagerEviction pod/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs Cancelling deletion of Pod opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8cgzs 7m4s Normal TaintManagerEviction pod/kserve-module-controller-manager-7566cf5fc8-ch2bw Cancelling deletion of Pod opendatahub/kserve-module-controller-manager-7566cf5fc8-ch2bw 6m53s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7779c685d8-d2f7f_46fc873b-1868-448c-9e90-51bb9c0e04b3 became leader