=== events in opendatahub (2026-11-03T16:45:11Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 22m Normal Scheduled pod/maas-controller-c4b666674-sbcxm Successfully assigned opendatahub/maas-controller-c4b666674-sbcxm to ip-10-0-129-122.us-east-2.compute.internal 31m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb to ip-10-0-129-122.us-east-2.compute.internal 29m Normal Scheduled pod/ai-gateway-controller-7cb8f6bb64-whsfj Successfully assigned opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj to ip-10-0-129-122.us-east-2.compute.internal 32m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-nk5ck to ip-10-0-133-129.us-east-2.compute.internal 21m Normal Scheduled pod/maas-controller-c4b666674-b7hjc Successfully assigned opendatahub/maas-controller-c4b666674-b7hjc to ip-10-0-129-122.us-east-2.compute.internal 31m Normal Scheduled pod/maas-controller-c4b666674-mnhsn Successfully assigned opendatahub/maas-controller-c4b666674-mnhsn to ip-10-0-129-122.us-east-2.compute.internal 33m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-lfxrp to ip-10-0-133-129.us-east-2.compute.internal 30m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-cljjs Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-cljjs to ip-10-0-133-129.us-east-2.compute.internal 32m Normal Scheduled pod/kserve-controller-manager-776cc66d47-ntj89 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-ntj89 to ip-10-0-129-122.us-east-2.compute.internal 29m Normal Scheduled pod/ai-gateway-controller-8c5c5b46c-bg6dh Successfully assigned opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh to ip-10-0-129-122.us-east-2.compute.internal 31m Normal Scheduled pod/maas-controller-579cf7c6d-mgwfv Successfully assigned opendatahub/maas-controller-579cf7c6d-mgwfv to ip-10-0-129-122.us-east-2.compute.internal 32m Normal Scheduled pod/odh-model-controller-769875db56-bjn8p Successfully assigned opendatahub/odh-model-controller-769875db56-bjn8p to ip-10-0-129-122.us-east-2.compute.internal 32m Normal Scheduled pod/model-serving-api-75bd89fdfc-j7zvw Successfully assigned opendatahub/model-serving-api-75bd89fdfc-j7zvw to ip-10-0-129-122.us-east-2.compute.internal 33m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 33m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-lfxrp 33m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 33m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Add eth0 [10.132.0.40/23] from ovn-kubernetes 33m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Container created 33m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.353s (2.353s including waiting). Image size: 195729000 bytes. 33m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-lfxrp Container started 32m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-lfxrp_7104ea32-768d-4930-be2e-92a0aa1d9ba1 became leader 32m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 32m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-ntj89 32m Normal Pulling pod/model-serving-api-75bd89fdfc-j7zvw Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 32m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 32m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-nk5ck 32m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 32m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-ntj89 Add eth0 [10.134.0.23/23] from ovn-kubernetes 32m Normal Pulling pod/odh-model-controller-769875db56-bjn8p Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 32m Normal AddedInterface pod/odh-model-controller-769875db56-bjn8p Add eth0 [10.134.0.25/23] from ovn-kubernetes 32m Normal AddedInterface pod/model-serving-api-75bd89fdfc-j7zvw Add eth0 [10.134.0.24/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-bjn8p 32m Normal Pulling pod/kserve-controller-manager-776cc66d47-ntj89 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 32m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 32m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Add eth0 [10.132.0.41/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-j7zvw 32m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 32m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 1.974s (1.974s including waiting). Image size: 235100981 bytes. 32m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Container created 32m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Container started 32m Normal Pulled pod/model-serving-api-75bd89fdfc-j7zvw Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.483s (2.483s including waiting). Image size: 184925761 bytes. 32m Normal Started pod/kserve-controller-manager-776cc66d47-ntj89 Container started 32m Normal Pulled pod/kserve-controller-manager-776cc66d47-ntj89 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.041s (3.041s including waiting). Image size: 247676581 bytes. 32m Normal Created pod/kserve-controller-manager-776cc66d47-ntj89 Container created 32m Normal Created pod/model-serving-api-75bd89fdfc-j7zvw Container created 32m Normal Started pod/model-serving-api-75bd89fdfc-j7zvw Container started 32m Normal Pulled pod/odh-model-controller-769875db56-bjn8p Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.41s (3.41s including waiting). Image size: 236755567 bytes. 32m Normal Created pod/odh-model-controller-769875db56-bjn8p Container created 32m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-bjn8p_0432eb7e-3920-4b85-ab04-224d30de6b1d became leader 32m Normal Started pod/odh-model-controller-769875db56-bjn8p Container started 32m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-ntj89_92a973d1-2e99-4d26-bca3-36babadd671d became leader 32m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-nk5ck_9f959e2f-7e87-4689-bf98-bee46c5a5641 became leader 31m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 31m Normal SuccessfulCreate replicaset/maas-controller-c4b666674 Created pod: maas-controller-c4b666674-mnhsn 31m Normal Pulling pod/maas-controller-579cf7c6d-mgwfv Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 31m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-mgwfv 31m Normal AddedInterface pod/maas-controller-579cf7c6d-mgwfv Add eth0 [10.134.0.29/23] from ovn-kubernetes 31m Normal AddedInterface pod/maas-controller-c4b666674-mnhsn Add eth0 [10.134.0.30/23] from ovn-kubernetes 31m Normal Pulling pod/maas-controller-c4b666674-mnhsn Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 31m Normal Created pod/maas-controller-579cf7c6d-mgwfv Container created 31m Normal Pulled pod/maas-controller-579cf7c6d-mgwfv Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.001s (3.001s including waiting). Image size: 200053271 bytes. 31m Normal Started pod/maas-controller-579cf7c6d-mgwfv Container started 31m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-mgwfv_5b32a67f-28a5-455a-9879-2305b2854840 became leader 31m Normal Started pod/maas-controller-c4b666674-mnhsn Container started 31m Normal Created pod/maas-controller-c4b666674-mnhsn Container created 31m Normal Pulled pod/maas-controller-c4b666674-mnhsn Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.924s (2.924s including waiting). Image size: 200053271 bytes. 31m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 31m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-4hdlb 31m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Add eth0 [10.134.0.31/23] from ovn-kubernetes 31m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 31m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 31m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-mgwfv 31m Normal Killing pod/maas-controller-579cf7c6d-mgwfv Stopping container manager 31m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Container started 31m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Container created 31m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 11.775s (11.775s including waiting). Image size: 1089502687 bytes. 31m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-c4b666674-mnhsn_577545e8-a9bc-4f9f-ba50-17987f738dde became leader 30m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-cljjs 30m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 30m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-cljjs Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 30m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-cljjs Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 430ms (430ms including waiting). Image size: 235100981 bytes. 30m Normal Created pod/llmisvc-controller-manager-6466c554d8-cljjs Container created 30m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-cljjs Add eth0 [10.132.0.43/23] from ovn-kubernetes 30m Normal Started pod/llmisvc-controller-manager-6466c554d8-cljjs Container started 30m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-nk5ck 30m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-nk5ck Stopping container manager 30m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 29m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-cljjs_9e6874bd-b786-437d-b87e-c565f676d126 became leader 29m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7cb8f6bb64 from 0 to 1 29m Normal SuccessfulCreate replicaset/ai-gateway-controller-7cb8f6bb64 Created pod: ai-gateway-controller-7cb8f6bb64-whsfj 29m Normal Pulling pod/ai-gateway-controller-7cb8f6bb64-whsfj Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" 29m Normal AddedInterface pod/ai-gateway-controller-7cb8f6bb64-whsfj Add eth0 [10.134.0.39/23] from ovn-kubernetes 29m Normal Created pod/ai-gateway-controller-7cb8f6bb64-whsfj Container created 29m Normal Started pod/ai-gateway-controller-7cb8f6bb64-whsfj Container started 29m Normal Pulled pod/ai-gateway-controller-7cb8f6bb64-whsfj Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" in 2.138s (2.138s including waiting). Image size: 183979434 bytes. 29m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7cb8f6bb64-whsfj_19858b2d-5df5-457f-bdba-b858baecedfe became leader 29m Normal SuccessfulCreate replicaset/ai-gateway-controller-8c5c5b46c Created pod: ai-gateway-controller-8c5c5b46c-bg6dh 29m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-8c5c5b46c from 0 to 1 29m Normal Pulling pod/ai-gateway-controller-8c5c5b46c-bg6dh Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" 29m Normal AddedInterface pod/ai-gateway-controller-8c5c5b46c-bg6dh Add eth0 [10.134.0.43/23] from ovn-kubernetes 29m Normal Started pod/ai-gateway-controller-8c5c5b46c-bg6dh Container started 29m Normal Created pod/ai-gateway-controller-8c5c5b46c-bg6dh Container created 29m Normal Pulled pod/ai-gateway-controller-8c5c5b46c-bg6dh Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" in 420ms (420ms including waiting). Image size: 183979434 bytes. 29m Normal Killing pod/ai-gateway-controller-7cb8f6bb64-whsfj Stopping container manager 29m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-7cb8f6bb64 from 1 to 0 29m Normal SuccessfulDelete replicaset/ai-gateway-controller-7cb8f6bb64 Deleted pod: ai-gateway-controller-7cb8f6bb64-whsfj 28m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-8c5c5b46c-bg6dh_3d3090f2-b367-4c6a-af8a-09e074c15495 became leader 23m Warning Unhealthy pod/model-serving-api-75bd89fdfc-j7zvw Readiness probe failed: Get "https://10.134.0.24:8443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 22m Normal Killing pod/maas-controller-c4b666674-mnhsn Stopping container manager 22m Normal SuccessfulDelete replicaset/maas-controller-c4b666674 Deleted pod: maas-controller-c4b666674-mnhsn 22m Normal SuccessfulCreate replicaset/maas-controller-c4b666674 Created pod: maas-controller-c4b666674-sbcxm 22m Normal Pulling pod/maas-controller-c4b666674-sbcxm Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 22m Normal AddedInterface pod/maas-controller-c4b666674-sbcxm Add eth0 [10.134.0.87/23] from ovn-kubernetes 22m Normal Started pod/maas-controller-c4b666674-sbcxm Container started 22m Normal Pulled pod/maas-controller-c4b666674-sbcxm Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 481ms (481ms including waiting). Image size: 200053271 bytes. 22m Normal Created pod/maas-controller-c4b666674-sbcxm Container created 22m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-c4b666674-sbcxm_bdd5132a-2d36-4e0e-b733-3458b94df8c3 became leader 21m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-c4b666674 from 1 to 0 21m Normal Killing pod/maas-controller-c4b666674-sbcxm Stopping container manager 21m Normal SuccessfulDelete replicaset/maas-controller-c4b666674 Deleted pod: maas-controller-c4b666674-sbcxm 21m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-c4b666674 from 0 to 1 21m Normal SuccessfulCreate replicaset/maas-controller-c4b666674 Created pod: maas-controller-c4b666674-b7hjc 21m Normal Pulling pod/maas-controller-c4b666674-b7hjc Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 21m Normal AddedInterface pod/maas-controller-c4b666674-b7hjc Add eth0 [10.134.0.89/23] from ovn-kubernetes 21m Normal Started pod/maas-controller-c4b666674-b7hjc Container started 21m Normal Created pod/maas-controller-c4b666674-b7hjc Container created 21m Normal Pulled pod/maas-controller-c4b666674-b7hjc Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 382ms (382ms including waiting). Image size: 200053271 bytes. 21m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-c4b666674-b7hjc_307028f5-36c8-4651-aaa5-6f0b8d20d576 became leader