=== events in opendatahub (2026-10-31T01:06:26Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 26m Normal Scheduled pod/maas-controller-7bdb48889c-shvsr Successfully assigned opendatahub/maas-controller-7bdb48889c-shvsr to ip-10-0-143-7.us-east-2.compute.internal 26m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls to ip-10-0-143-188.us-east-2.compute.internal 25m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-hzz82 Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-hzz82 to ip-10-0-143-188.us-east-2.compute.internal 26m Normal Scheduled pod/maas-controller-579cf7c6d-28vzm Successfully assigned opendatahub/maas-controller-579cf7c6d-28vzm to ip-10-0-143-7.us-east-2.compute.internal 17m Normal Scheduled pod/maas-controller-7bdb48889c-rchq7 Successfully assigned opendatahub/maas-controller-7bdb48889c-rchq7 to ip-10-0-143-7.us-east-2.compute.internal 26m Normal Scheduled pod/kserve-controller-manager-776cc66d47-wvk8t Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-wvk8t to ip-10-0-143-7.us-east-2.compute.internal 26m Normal Scheduled pod/model-serving-api-75bd89fdfc-xznzv Successfully assigned opendatahub/model-serving-api-75bd89fdfc-xznzv to ip-10-0-143-7.us-east-2.compute.internal 24m Normal Scheduled pod/ai-gateway-controller-697b76b867-g76pk Successfully assigned opendatahub/ai-gateway-controller-697b76b867-g76pk to ip-10-0-143-7.us-east-2.compute.internal 26m Normal Scheduled pod/odh-model-controller-769875db56-k8s5d Successfully assigned opendatahub/odh-model-controller-769875db56-k8s5d to ip-10-0-143-7.us-east-2.compute.internal 26m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg to ip-10-0-143-188.us-east-2.compute.internal 23m Normal Scheduled pod/ai-gateway-controller-5bb95b6d85-g6jkp Successfully assigned opendatahub/ai-gateway-controller-5bb95b6d85-g6jkp to ip-10-0-142-217.us-east-2.compute.internal 16m Normal Scheduled pod/maas-controller-7bdb48889c-wcwl9 Successfully assigned opendatahub/maas-controller-7bdb48889c-wcwl9 to ip-10-0-143-7.us-east-2.compute.internal 27m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h to ip-10-0-143-188.us-east-2.compute.internal 27m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 27m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-8kw7h 27m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Add eth0 [10.133.0.23/23] from ovn-kubernetes 27m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 27m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.592s (1.592s including waiting). Image size: 195729000 bytes. 27m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Container created 27m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-8kw7h Container started 27m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-8kw7h_6217c9e8-5257-4f41-8d29-278dfc6a7f0e became leader 26m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-28vzm 26m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-xznzv 26m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-sdqcg 26m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 26m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 26m Normal Pulling pod/maas-controller-579cf7c6d-28vzm Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 26m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-wvk8t 26m Normal AddedInterface pod/maas-controller-579cf7c6d-28vzm Add eth0 [10.134.0.31/23] from ovn-kubernetes 26m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 26m Normal SuccessfulCreate replicaset/maas-controller-7bdb48889c Created pod: maas-controller-7bdb48889c-shvsr 26m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 26m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-k8s5d 26m Warning FailedMount pod/odh-model-controller-769875db56-k8s5d MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 26m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 26m Normal AddedInterface pod/maas-controller-7bdb48889c-shvsr Add eth0 [10.134.0.35/23] from ovn-kubernetes 26m Normal Pulling pod/model-serving-api-75bd89fdfc-xznzv Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 26m Normal AddedInterface pod/odh-model-controller-769875db56-k8s5d Add eth0 [10.134.0.34/23] from ovn-kubernetes 26m Normal Pulling pod/odh-model-controller-769875db56-k8s5d Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 26m Normal AddedInterface pod/model-serving-api-75bd89fdfc-xznzv Add eth0 [10.134.0.33/23] from ovn-kubernetes 26m Normal Pulling pod/maas-controller-7bdb48889c-shvsr Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 26m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-wvk8t Add eth0 [10.134.0.32/23] from ovn-kubernetes 26m Normal Pulling pod/kserve-controller-manager-776cc66d47-wvk8t Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 26m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 26m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Add eth0 [10.133.0.25/23] from ovn-kubernetes 26m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Container started 26m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Container created 26m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.041s (2.041s including waiting). Image size: 235100981 bytes. 26m Normal Created pod/maas-controller-579cf7c6d-28vzm Container created 26m Normal Started pod/kserve-controller-manager-776cc66d47-wvk8t Container started 26m Normal Pulled pod/model-serving-api-75bd89fdfc-xznzv Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 3.944s (3.944s including waiting). Image size: 184925761 bytes. 26m Normal Created pod/model-serving-api-75bd89fdfc-xznzv Container created 26m Normal Started pod/model-serving-api-75bd89fdfc-xznzv Container started 26m Normal Started pod/maas-controller-7bdb48889c-shvsr Container started 26m Normal Created pod/maas-controller-7bdb48889c-shvsr Container created 26m Normal Pulled pod/maas-controller-7bdb48889c-shvsr Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 4.575s (4.575s including waiting). Image size: 200053271 bytes. 26m Normal Pulled pod/kserve-controller-manager-776cc66d47-wvk8t Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.977s (3.977s including waiting). Image size: 247676581 bytes. 26m Normal Started pod/maas-controller-579cf7c6d-28vzm Container started 26m Normal Created pod/kserve-controller-manager-776cc66d47-wvk8t Container created 26m Normal Pulled pod/maas-controller-579cf7c6d-28vzm Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 4.702s (4.702s including waiting). Image size: 200053271 bytes. 26m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-sdqcg_09385aa9-5a1d-4041-abab-a62bd1799852 became leader 26m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-wvk8t_5e7c7373-d2d9-459a-a9f6-5aae9c734677 became leader 26m Normal Created pod/odh-model-controller-769875db56-k8s5d Container created 26m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-k8s5d_62fd2086-3a73-4cc9-a19e-14982c8a15ac became leader 26m Normal Pulled pod/odh-model-controller-769875db56-k8s5d Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 4.279s (4.279s including waiting). Image size: 236755567 bytes. 26m Normal Started pod/odh-model-controller-769875db56-k8s5d Container started 26m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-28vzm_c1211c75-e38d-4cd4-9ea9-d072b1607207 became leader 26m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 26m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 26m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-hkvls 26m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Add eth0 [10.133.0.27/23] from ovn-kubernetes 26m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 26m Normal Killing pod/maas-controller-579cf7c6d-28vzm Stopping container manager 26m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-28vzm 26m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Container created 26m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 11.366s (11.366s including waiting). Image size: 1089502687 bytes. 26m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-hkvls Container started 26m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-7bdb48889c-shvsr_d20d2a26-c364-4497-925e-e130308d4187 became leader 25m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 25m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-hzz82 25m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-hzz82 Add eth0 [10.133.0.29/23] from ovn-kubernetes 25m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-hzz82 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 25m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-hzz82 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 431ms (431ms including waiting). Image size: 235100981 bytes. 25m Normal Created pod/llmisvc-controller-manager-6466c554d8-hzz82 Container created 25m Normal Started pod/llmisvc-controller-manager-6466c554d8-hzz82 Container started 25m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-sdqcg 25m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-sdqcg Stopping container manager 25m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 24m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-hzz82_93ade751-6cc3-45da-8e0c-daffd9525142 became leader 24m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-697b76b867 from 0 to 1 24m Normal SuccessfulCreate replicaset/ai-gateway-controller-697b76b867 Created pod: ai-gateway-controller-697b76b867-g76pk 24m Normal AddedInterface pod/ai-gateway-controller-697b76b867-g76pk Add eth0 [10.134.0.44/23] from ovn-kubernetes 24m Normal Pulling pod/ai-gateway-controller-697b76b867-g76pk Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" 24m Normal Created pod/ai-gateway-controller-697b76b867-g76pk Container created 24m Normal Pulled pod/ai-gateway-controller-697b76b867-g76pk Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" in 3.861s (3.861s including waiting). Image size: 183979424 bytes. 24m Normal Started pod/ai-gateway-controller-697b76b867-g76pk Container started 24m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-697b76b867-g76pk_c3591ccd-f520-4235-8a04-0e182adadc9c became leader 23m Normal SuccessfulCreate replicaset/ai-gateway-controller-5bb95b6d85 Created pod: ai-gateway-controller-5bb95b6d85-g6jkp 23m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5bb95b6d85 from 0 to 1 23m Normal Pulling pod/ai-gateway-controller-5bb95b6d85-g6jkp Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" 23m Normal AddedInterface pod/ai-gateway-controller-5bb95b6d85-g6jkp Add eth0 [10.132.0.55/23] from ovn-kubernetes 23m Normal Started pod/ai-gateway-controller-5bb95b6d85-g6jkp Container started 23m Normal Created pod/ai-gateway-controller-5bb95b6d85-g6jkp Container created 23m Normal Pulled pod/ai-gateway-controller-5bb95b6d85-g6jkp Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" in 2.27s (2.27s including waiting). Image size: 183979424 bytes. 23m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-697b76b867 from 1 to 0 23m Normal SuccessfulDelete replicaset/ai-gateway-controller-697b76b867 Deleted pod: ai-gateway-controller-697b76b867-g76pk 23m Normal Killing pod/ai-gateway-controller-697b76b867-g76pk Stopping container manager 23m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5bb95b6d85-g6jkp_e95bc73b-f235-4f21-8cff-3600ba7ab276 became leader 18m Warning Unhealthy pod/model-serving-api-75bd89fdfc-xznzv Liveness probe failed: Get "https://10.134.0.33:8443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 17m Normal Killing pod/maas-controller-7bdb48889c-shvsr Stopping container manager 17m Normal SuccessfulDelete replicaset/maas-controller-7bdb48889c Deleted pod: maas-controller-7bdb48889c-shvsr 17m Normal SuccessfulCreate replicaset/maas-controller-7bdb48889c Created pod: maas-controller-7bdb48889c-rchq7 17m Normal Created pod/maas-controller-7bdb48889c-rchq7 Container created 17m Normal Pulling pod/maas-controller-7bdb48889c-rchq7 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 17m Normal Pulled pod/maas-controller-7bdb48889c-rchq7 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 472ms (472ms including waiting). Image size: 200053271 bytes. 17m Normal Started pod/maas-controller-7bdb48889c-rchq7 Container started 17m Normal AddedInterface pod/maas-controller-7bdb48889c-rchq7 Add eth0 [10.134.0.80/23] from ovn-kubernetes 16m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-7bdb48889c-rchq7_7d99741d-37e3-4522-b7ef-2749ee2426d6 became leader 16m Normal SuccessfulDelete replicaset/maas-controller-7bdb48889c Deleted pod: maas-controller-7bdb48889c-rchq7 16m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-7bdb48889c from 1 to 0 16m Normal Killing pod/maas-controller-7bdb48889c-rchq7 Stopping container manager 16m Normal SuccessfulCreate replicaset/maas-controller-7bdb48889c Created pod: maas-controller-7bdb48889c-wcwl9 16m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-7bdb48889c from 0 to 1 16m Normal AddedInterface pod/maas-controller-7bdb48889c-wcwl9 Add eth0 [10.134.0.81/23] from ovn-kubernetes 16m Normal Pulling pod/maas-controller-7bdb48889c-wcwl9 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 16m Normal Pulled pod/maas-controller-7bdb48889c-wcwl9 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 491ms (491ms including waiting). Image size: 200053271 bytes. 16m Normal Created pod/maas-controller-7bdb48889c-wcwl9 Container created 16m Normal Started pod/maas-controller-7bdb48889c-wcwl9 Container started 16m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-7bdb48889c-wcwl9_cf0ac6b8-e04f-4a28-aea2-9e399ab2936c became leader