=== events in opendatahub (2026-10-30T21:24:31Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 21m Normal Scheduled pod/maas-controller-54df5bbf78-pnpz5 Successfully assigned opendatahub/maas-controller-54df5bbf78-pnpz5 to ip-10-0-135-104.us-east-2.compute.internal 38m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk to ip-10-0-141-65.us-east-2.compute.internal 28m Normal Scheduled pod/test-curl-7100-489caf Successfully assigned opendatahub/test-curl-7100-489caf to ip-10-0-135-104.us-east-2.compute.internal 27m Normal Scheduled pod/test-curl-7100-2bba7e Successfully assigned opendatahub/test-curl-7100-2bba7e to ip-10-0-135-104.us-east-2.compute.internal 39m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-9zt2k to ip-10-0-135-104.us-east-2.compute.internal 37m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-sdjpb Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb to ip-10-0-141-65.us-east-2.compute.internal 38m Normal Scheduled pod/kserve-controller-manager-776cc66d47-c7jm5 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-c7jm5 to ip-10-0-135-104.us-east-2.compute.internal 35m Normal Scheduled pod/ai-gateway-controller-5f6dd4f69f-k26sl Successfully assigned opendatahub/ai-gateway-controller-5f6dd4f69f-k26sl to ip-10-0-135-104.us-east-2.compute.internal 38m Normal Scheduled pod/maas-controller-579cf7c6d-ztzrs Successfully assigned opendatahub/maas-controller-579cf7c6d-ztzrs to ip-10-0-135-104.us-east-2.compute.internal 38m Normal Scheduled pod/odh-model-controller-769875db56-7kxpw Successfully assigned opendatahub/odh-model-controller-769875db56-7kxpw to ip-10-0-135-104.us-east-2.compute.internal 37m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 to ip-10-0-135-104.us-east-2.compute.internal 20m Normal Scheduled pod/maas-controller-54df5bbf78-sjqgq Successfully assigned opendatahub/maas-controller-54df5bbf78-sjqgq to ip-10-0-135-104.us-east-2.compute.internal 35m Normal Scheduled pod/ai-gateway-controller-5d4f8585d4-9vb62 Successfully assigned opendatahub/ai-gateway-controller-5d4f8585d4-9vb62 to ip-10-0-135-104.us-east-2.compute.internal 38m Normal Scheduled pod/maas-controller-54df5bbf78-7m69q Successfully assigned opendatahub/maas-controller-54df5bbf78-7m69q to ip-10-0-135-104.us-east-2.compute.internal 38m Normal Scheduled pod/model-serving-api-75bd89fdfc-r7zmp Successfully assigned opendatahub/model-serving-api-75bd89fdfc-r7zmp to ip-10-0-135-104.us-east-2.compute.internal 39m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Add eth0 [10.132.0.46/23] from ovn-kubernetes 39m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 39m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-9zt2k 39m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 39m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.405s (2.405s including waiting). Image size: 195729000 bytes. 39m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Container started 39m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-9zt2k Container created 39m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-9zt2k_99b168eb-d0e4-4175-b0c0-61e451f95386 became leader 38m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 38m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 38m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-r7zmp 38m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-c7jm5 38m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 38m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-7kxpw 38m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 38m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-4q2hk 38m Normal AddedInterface pod/odh-model-controller-769875db56-7kxpw Add eth0 [10.132.0.52/23] from ovn-kubernetes 38m Normal Pulling pod/kserve-controller-manager-776cc66d47-c7jm5 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 38m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Add eth0 [10.134.0.27/23] from ovn-kubernetes 38m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-c7jm5 Add eth0 [10.132.0.50/23] from ovn-kubernetes 38m Normal AddedInterface pod/model-serving-api-75bd89fdfc-r7zmp Add eth0 [10.132.0.51/23] from ovn-kubernetes 38m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 38m Normal Pulling pod/model-serving-api-75bd89fdfc-r7zmp Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 38m Normal Pulling pod/odh-model-controller-769875db56-7kxpw Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 38m Normal Pulled pod/model-serving-api-75bd89fdfc-r7zmp Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.454s (2.454s including waiting). Image size: 184925761 bytes. 38m Normal Pulled pod/kserve-controller-manager-776cc66d47-c7jm5 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.77s (2.77s including waiting). Image size: 247676581 bytes. 38m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Container started 38m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Container created 38m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 1.982s (1.983s including waiting). Image size: 235100981 bytes. 38m Normal Created pod/kserve-controller-manager-776cc66d47-c7jm5 Container created 38m Normal Started pod/kserve-controller-manager-776cc66d47-c7jm5 Container started 38m Normal Started pod/model-serving-api-75bd89fdfc-r7zmp Container started 38m Normal Created pod/model-serving-api-75bd89fdfc-r7zmp Container created 38m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-c7jm5_f662264e-5463-4661-9f4c-9375ccf27b7c became leader 38m Normal Pulled pod/odh-model-controller-769875db56-7kxpw Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.326s (3.326s including waiting). Image size: 236755567 bytes. 38m Normal Created pod/odh-model-controller-769875db56-7kxpw Container created 38m Normal Started pod/odh-model-controller-769875db56-7kxpw Container started 38m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-7kxpw_02f5bfab-4111-4f07-b3d4-709c363b439b became leader 38m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-4q2hk_11a52d99-b526-4684-a0de-ffb24d8297fc became leader 38m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 38m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-ztzrs 38m Normal AddedInterface pod/maas-controller-54df5bbf78-7m69q Add eth0 [10.132.0.55/23] from ovn-kubernetes 38m Normal Pulling pod/maas-controller-579cf7c6d-ztzrs Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 38m Normal AddedInterface pod/maas-controller-579cf7c6d-ztzrs Add eth0 [10.132.0.54/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/maas-controller-54df5bbf78 Created pod: maas-controller-54df5bbf78-7m69q 38m Normal Pulling pod/maas-controller-54df5bbf78-7m69q Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 37m Normal Started pod/maas-controller-54df5bbf78-7m69q Container started 37m Normal Started pod/maas-controller-579cf7c6d-ztzrs Container started 37m Normal Pulled pod/maas-controller-54df5bbf78-7m69q Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.629s (2.629s including waiting). Image size: 200053271 bytes. 37m Normal Created pod/maas-controller-54df5bbf78-7m69q Container created 37m Normal Pulled pod/maas-controller-579cf7c6d-ztzrs Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.607s (2.607s including waiting). Image size: 200053271 bytes. 37m Normal Created pod/maas-controller-579cf7c6d-ztzrs Container created 37m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 37m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 37m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-ztzrs_75a18e79-3470-401e-80aa-3e579dca56ed became leader 37m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 37m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Add eth0 [10.132.0.56/23] from ovn-kubernetes 37m Normal Killing pod/maas-controller-579cf7c6d-ztzrs Stopping container manager 37m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-ztzrs 37m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 37m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Container started 37m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Container created 37m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-bq4f9 Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 11.414s (11.414s including waiting). Image size: 1089502687 bytes. 37m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-54df5bbf78-7m69q_1794a927-02b0-4751-974e-8c50f6d09c6d became leader 37m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 37m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-sdjpb 37m Normal Created pod/llmisvc-controller-manager-6466c554d8-sdjpb Container created 37m Normal Started pod/llmisvc-controller-manager-6466c554d8-sdjpb Container started 37m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-sdjpb Add eth0 [10.134.0.29/23] from ovn-kubernetes 37m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-sdjpb Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 37m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-sdjpb Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 335ms (335ms including waiting). Image size: 235100981 bytes. 36m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-4q2hk Stopping container manager 36m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 36m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-4q2hk 36m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-sdjpb_5df732c9-7c88-4fef-b5a9-7f7425d99009 became leader 35m Normal SuccessfulCreate replicaset/ai-gateway-controller-5f6dd4f69f Created pod: ai-gateway-controller-5f6dd4f69f-k26sl 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5f6dd4f69f from 0 to 1 35m Normal AddedInterface pod/ai-gateway-controller-5f6dd4f69f-k26sl Add eth0 [10.132.0.64/23] from ovn-kubernetes 35m Normal Pulling pod/ai-gateway-controller-5f6dd4f69f-k26sl Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-193" 35m Normal Started pod/ai-gateway-controller-5f6dd4f69f-k26sl Container started 35m Normal Pulled pod/ai-gateway-controller-5f6dd4f69f-k26sl Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-193" in 2.254s (2.254s including waiting). Image size: 183926676 bytes. 35m Normal Created pod/ai-gateway-controller-5f6dd4f69f-k26sl Container created 35m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5f6dd4f69f-k26sl_87f6f71b-12a7-4d32-9231-917ed54e1524 became leader 35m Normal SuccessfulCreate replicaset/ai-gateway-controller-5d4f8585d4 Created pod: ai-gateway-controller-5d4f8585d4-9vb62 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5d4f8585d4 from 0 to 1 35m Normal Started pod/ai-gateway-controller-5d4f8585d4-9vb62 Container started 35m Normal Created pod/ai-gateway-controller-5d4f8585d4-9vb62 Container created 35m Normal Pulled pod/ai-gateway-controller-5d4f8585d4-9vb62 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-193" in 364ms (364ms including waiting). Image size: 183926676 bytes. 35m Normal Pulling pod/ai-gateway-controller-5d4f8585d4-9vb62 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-193" 35m Normal AddedInterface pod/ai-gateway-controller-5d4f8585d4-9vb62 Add eth0 [10.132.0.67/23] from ovn-kubernetes 35m Normal SuccessfulDelete replicaset/ai-gateway-controller-5f6dd4f69f Deleted pod: ai-gateway-controller-5f6dd4f69f-k26sl 35m Normal Killing pod/ai-gateway-controller-5f6dd4f69f-k26sl Stopping container manager 35m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5f6dd4f69f from 1 to 0 35m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5d4f8585d4-9vb62_931a7f41-ce4e-4f5f-b70e-fcc1df5b4377 became leader 28m Normal AddedInterface pod/test-curl-7100-489caf Add eth0 [10.132.0.110/23] from ovn-kubernetes 28m Normal Pulling pod/test-curl-7100-489caf Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 28m Normal Pulled pod/test-curl-7100-489caf Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 692ms (692ms including waiting). Image size: 106100986 bytes. 28m Normal Started pod/test-curl-7100-489caf Container started 28m Normal Created pod/test-curl-7100-489caf Container created 28m Warning Unhealthy pod/model-serving-api-75bd89fdfc-r7zmp Readiness probe failed: Get "https://10.132.0.51:8443/readyz": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 27m Normal Killing pod/test-curl-7100-489caf Stopping container test-curl-7100-489caf 27m Normal Started pod/test-curl-7100-2bba7e Container started 27m Normal AddedInterface pod/test-curl-7100-2bba7e Add eth0 [10.132.0.116/23] from ovn-kubernetes 27m Normal Pulled pod/test-curl-7100-2bba7e Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 27m Normal Created pod/test-curl-7100-2bba7e Container created 27m Normal Killing pod/test-curl-7100-2bba7e Stopping container test-curl-7100-2bba7e 21m Normal SuccessfulDelete replicaset/maas-controller-54df5bbf78 Deleted pod: maas-controller-54df5bbf78-7m69q 21m Normal Killing pod/maas-controller-54df5bbf78-7m69q Stopping container manager 21m Normal AddedInterface pod/maas-controller-54df5bbf78-pnpz5 Add eth0 [10.132.0.126/23] from ovn-kubernetes 21m Normal SuccessfulCreate replicaset/maas-controller-54df5bbf78 Created pod: maas-controller-54df5bbf78-pnpz5 21m Normal Pulling pod/maas-controller-54df5bbf78-pnpz5 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 21m Normal Pulled pod/maas-controller-54df5bbf78-pnpz5 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 490ms (490ms including waiting). Image size: 200053271 bytes. 21m Normal Created pod/maas-controller-54df5bbf78-pnpz5 Container created 21m Normal Started pod/maas-controller-54df5bbf78-pnpz5 Container started 21m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-54df5bbf78-pnpz5_87bc6181-69aa-4448-ab9e-d46756d5312f became leader 20m Normal Killing pod/maas-controller-54df5bbf78-pnpz5 Stopping container manager 20m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-54df5bbf78 from 1 to 0 20m Normal SuccessfulDelete replicaset/maas-controller-54df5bbf78 Deleted pod: maas-controller-54df5bbf78-pnpz5 20m Normal SuccessfulCreate replicaset/maas-controller-54df5bbf78 Created pod: maas-controller-54df5bbf78-sjqgq 20m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-54df5bbf78 from 0 to 1 20m Normal AddedInterface pod/maas-controller-54df5bbf78-sjqgq Add eth0 [10.132.0.127/23] from ovn-kubernetes 20m Normal Pulling pod/maas-controller-54df5bbf78-sjqgq Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 20m Normal Pulled pod/maas-controller-54df5bbf78-sjqgq Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 449ms (449ms including waiting). Image size: 200053271 bytes. 20m Normal Created pod/maas-controller-54df5bbf78-sjqgq Container created 20m Normal Started pod/maas-controller-54df5bbf78-sjqgq Container started 20m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-54df5bbf78-sjqgq_05309c05-1e73-4e70-b1d2-55455f2884a5 became leader