=== events in opendatahub (2026-11-02T18:20:17Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 34m Normal Scheduled pod/maas-controller-579cf7c6d-cnlqh Successfully assigned opendatahub/maas-controller-579cf7c6d-cnlqh to ip-10-0-135-9.us-east-2.compute.internal 18m Normal Scheduled pod/maas-controller-698c564c8b-rgrcd Successfully assigned opendatahub/maas-controller-698c564c8b-rgrcd to ip-10-0-135-9.us-east-2.compute.internal 25m Normal Scheduled pod/test-curl-7719-b76500 Successfully assigned opendatahub/test-curl-7719-b76500 to ip-10-0-132-5.us-east-2.compute.internal 25m Normal Scheduled pod/test-curl-7716-e49fca Successfully assigned opendatahub/test-curl-7716-e49fca to ip-10-0-135-9.us-east-2.compute.internal 33m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-ckhld Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-ckhld to ip-10-0-141-84.us-east-2.compute.internal 34m Normal Scheduled pod/kserve-controller-manager-776cc66d47-4pwjv Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-4pwjv to ip-10-0-135-9.us-east-2.compute.internal 34m Normal Scheduled pod/odh-model-controller-769875db56-5ncdw Successfully assigned opendatahub/odh-model-controller-769875db56-5ncdw to ip-10-0-135-9.us-east-2.compute.internal 34m Normal Scheduled pod/maas-controller-698c564c8b-v5qgk Successfully assigned opendatahub/maas-controller-698c564c8b-v5qgk to ip-10-0-135-9.us-east-2.compute.internal 34m Normal Scheduled pod/model-serving-api-75bd89fdfc-5x726 Successfully assigned opendatahub/model-serving-api-75bd89fdfc-5x726 to ip-10-0-135-9.us-east-2.compute.internal 32m Normal Scheduled pod/ai-gateway-controller-fdd7ff9-r2xj6 Successfully assigned opendatahub/ai-gateway-controller-fdd7ff9-r2xj6 to ip-10-0-135-9.us-east-2.compute.internal 34m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt to ip-10-0-135-9.us-east-2.compute.internal 32m Normal Scheduled pod/ai-gateway-controller-b59d66cb5-2vkbv Successfully assigned opendatahub/ai-gateway-controller-b59d66cb5-2vkbv to ip-10-0-135-9.us-east-2.compute.internal 35m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx to ip-10-0-135-9.us-east-2.compute.internal 34m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-sdrxn to ip-10-0-141-84.us-east-2.compute.internal 18m Normal Scheduled pod/maas-controller-698c564c8b-fvvw4 Successfully assigned opendatahub/maas-controller-698c564c8b-fvvw4 to ip-10-0-135-9.us-east-2.compute.internal 35m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 35m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-rxjvx 35m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Add eth0 [10.133.0.23/23] from ovn-kubernetes 35m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 35m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.554s (1.554s including waiting). Image size: 195729000 bytes. 35m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Container started 35m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-rxjvx Container created 35m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-rxjvx_9b439b1b-a55d-48eb-9531-3672b0dfbeae became leader 34m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-5x726 34m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-4pwjv Add eth0 [10.133.0.26/23] from ovn-kubernetes 34m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Add eth0 [10.134.0.29/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-sdrxn 34m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-4pwjv 34m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 34m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 34m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 34m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-5ncdw 34m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 34m Normal Pulling pod/model-serving-api-75bd89fdfc-5x726 Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 34m Normal AddedInterface pod/model-serving-api-75bd89fdfc-5x726 Add eth0 [10.133.0.27/23] from ovn-kubernetes 34m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 34m Normal AddedInterface pod/odh-model-controller-769875db56-5ncdw Add eth0 [10.133.0.28/23] from ovn-kubernetes 34m Normal Pulling pod/odh-model-controller-769875db56-5ncdw Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 34m Normal Pulling pod/kserve-controller-manager-776cc66d47-4pwjv Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 34m Normal Started pod/model-serving-api-75bd89fdfc-5x726 Container started 34m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Container started 34m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Container created 34m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.412s (2.412s including waiting). Image size: 235100981 bytes. 34m Normal Pulled pod/model-serving-api-75bd89fdfc-5x726 Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.69s (2.69s including waiting). Image size: 184925761 bytes. 34m Normal Created pod/model-serving-api-75bd89fdfc-5x726 Container created 34m Normal Created pod/odh-model-controller-769875db56-5ncdw Container created 34m Normal Created pod/kserve-controller-manager-776cc66d47-4pwjv Container created 34m Normal Pulled pod/kserve-controller-manager-776cc66d47-4pwjv Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.008s (3.008s including waiting). Image size: 247676581 bytes. 34m Normal Pulled pod/odh-model-controller-769875db56-5ncdw Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.57s (3.57s including waiting). Image size: 236755567 bytes. 34m Normal Started pod/kserve-controller-manager-776cc66d47-4pwjv Container started 34m Normal Started pod/odh-model-controller-769875db56-5ncdw Container started 34m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-5ncdw_a214e5f0-492e-490a-b02c-6ad6b461e2f9 became leader 34m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-4pwjv_f9840d33-ed1a-4d5e-a45e-54e7aeddf6e9 became leader 34m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-sdrxn_b5340ad1-7252-4fc1-ae56-ccb84bcc0388 became leader 34m Normal Pulling pod/maas-controller-579cf7c6d-cnlqh Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 34m Normal AddedInterface pod/maas-controller-579cf7c6d-cnlqh Add eth0 [10.133.0.30/23] from ovn-kubernetes 34m Normal AddedInterface pod/maas-controller-698c564c8b-v5qgk Add eth0 [10.133.0.31/23] from ovn-kubernetes 34m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 34m Normal SuccessfulCreate replicaset/maas-controller-698c564c8b Created pod: maas-controller-698c564c8b-v5qgk 34m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-cnlqh 34m Normal Pulling pod/maas-controller-698c564c8b-v5qgk Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 34m Normal Started pod/maas-controller-698c564c8b-v5qgk Container started 34m Normal Pulled pod/maas-controller-698c564c8b-v5qgk Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.799s (2.799s including waiting). Image size: 200053271 bytes. 34m Normal Pulled pod/maas-controller-579cf7c6d-cnlqh Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.981s (2.981s including waiting). Image size: 200053271 bytes. 34m Normal Created pod/maas-controller-579cf7c6d-cnlqh Container created 34m Normal Started pod/maas-controller-579cf7c6d-cnlqh Container started 34m Normal Created pod/maas-controller-698c564c8b-v5qgk Container created 34m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-cnlqh_4f306113-35c2-4245-93e5-edb3606c5dbe became leader 34m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 34m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-8twlt 34m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 34m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Add eth0 [10.133.0.32/23] from ovn-kubernetes 34m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-cnlqh 34m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 34m Normal Killing pod/maas-controller-579cf7c6d-cnlqh Stopping container manager 34m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Container created 34m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.872s (10.872s including waiting). Image size: 1089502687 bytes. 34m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Container started 34m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-698c564c8b-v5qgk_a621a05a-ed00-4e66-bf5a-6f1cfdc82623 became leader 33m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 33m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-ckhld 33m Normal Started pod/llmisvc-controller-manager-6466c554d8-ckhld Container started 33m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-ckhld Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 400ms (400ms including waiting). Image size: 235100981 bytes. 33m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-ckhld Add eth0 [10.134.0.32/23] from ovn-kubernetes 33m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-ckhld Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 33m Normal Created pod/llmisvc-controller-manager-6466c554d8-ckhld Container created 33m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-sdrxn 33m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-sdrxn Stopping container manager 33m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 32m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-ckhld_5bdb00f9-9b19-4985-8619-221097a4e652 became leader 32m Normal SuccessfulCreate replicaset/ai-gateway-controller-b59d66cb5 Created pod: ai-gateway-controller-b59d66cb5-2vkbv 32m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-b59d66cb5 from 0 to 1 32m Normal AddedInterface pod/ai-gateway-controller-b59d66cb5-2vkbv Add eth0 [10.133.0.37/23] from ovn-kubernetes 32m Normal Pulling pod/ai-gateway-controller-b59d66cb5-2vkbv Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" 32m Normal Pulled pod/ai-gateway-controller-b59d66cb5-2vkbv Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" in 4.007s (4.007s including waiting). Image size: 183983521 bytes. 32m Normal Created pod/ai-gateway-controller-b59d66cb5-2vkbv Container created 32m Normal Started pod/ai-gateway-controller-b59d66cb5-2vkbv Container started 32m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-b59d66cb5-2vkbv_b7032e9f-dcbb-4da6-8ecf-20a6032b7d25 became leader 32m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-fdd7ff9 from 0 to 1 32m Normal AddedInterface pod/ai-gateway-controller-fdd7ff9-r2xj6 Add eth0 [10.133.0.39/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/ai-gateway-controller-fdd7ff9 Created pod: ai-gateway-controller-fdd7ff9-r2xj6 32m Normal Pulling pod/ai-gateway-controller-fdd7ff9-r2xj6 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" 32m Normal Created pod/ai-gateway-controller-fdd7ff9-r2xj6 Container created 32m Normal Pulled pod/ai-gateway-controller-fdd7ff9-r2xj6 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-209" in 438ms (438ms including waiting). Image size: 183983521 bytes. 32m Normal Started pod/ai-gateway-controller-fdd7ff9-r2xj6 Container started 31m Normal SuccessfulDelete replicaset/ai-gateway-controller-b59d66cb5 Deleted pod: ai-gateway-controller-b59d66cb5-2vkbv 31m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-b59d66cb5 from 1 to 0 31m Normal Killing pod/ai-gateway-controller-b59d66cb5-2vkbv Stopping container manager 31m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-fdd7ff9-r2xj6_6872cbb6-b389-44e7-a2c6-9272a3136e86 became leader 27m Warning Unhealthy pod/maas-controller-698c564c8b-v5qgk Readiness probe failed: Get "http://10.133.0.31:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-8twlt Readiness probe failed: Get "https://10.133.0.32:8443/ready": context deadline exceeded 25m Normal AddedInterface pod/test-curl-7719-b76500 Add eth0 [10.132.0.97/23] from ovn-kubernetes 25m Normal Pulling pod/test-curl-7719-b76500 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 25m Normal AddedInterface pod/test-curl-7716-e49fca Add eth0 [10.133.0.133/23] from ovn-kubernetes 25m Normal Pulled pod/test-curl-7719-b76500 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.587s (2.587s including waiting). Image size: 106100986 bytes. 25m Normal Pulling pod/test-curl-7716-e49fca Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 25m Normal Started pod/test-curl-7716-e49fca Container started 25m Normal Started pod/test-curl-7719-b76500 Container started 25m Normal Created pod/test-curl-7719-b76500 Container created 25m Normal Pulled pod/test-curl-7716-e49fca Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 732ms (732ms including waiting). Image size: 106100986 bytes. 25m Normal Created pod/test-curl-7716-e49fca Container created 25m Normal Killing pod/test-curl-7719-b76500 Stopping container test-curl-7719-b76500 25m Normal Killing pod/test-curl-7716-e49fca Stopping container test-curl-7716-e49fca 18m Normal Killing pod/maas-controller-698c564c8b-v5qgk Stopping container manager 18m Normal SuccessfulDelete replicaset/maas-controller-698c564c8b Deleted pod: maas-controller-698c564c8b-v5qgk 18m Normal AddedInterface pod/maas-controller-698c564c8b-fvvw4 Add eth0 [10.133.0.178/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/maas-controller-698c564c8b Created pod: maas-controller-698c564c8b-fvvw4 18m Normal Pulling pod/maas-controller-698c564c8b-fvvw4 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal Created pod/maas-controller-698c564c8b-fvvw4 Container created 18m Normal Pulled pod/maas-controller-698c564c8b-fvvw4 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 524ms (524ms including waiting). Image size: 200053271 bytes. 18m Normal Started pod/maas-controller-698c564c8b-fvvw4 Container started 18m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-698c564c8b-fvvw4_a3ff0788-c681-4923-8cb3-1cd29228a881 became leader 18m Normal Killing pod/maas-controller-698c564c8b-fvvw4 Stopping container manager 18m Normal SuccessfulDelete replicaset/maas-controller-698c564c8b Deleted pod: maas-controller-698c564c8b-fvvw4 18m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-698c564c8b from 1 to 0 18m Normal Pulling pod/maas-controller-698c564c8b-rgrcd Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal AddedInterface pod/maas-controller-698c564c8b-rgrcd Add eth0 [10.133.0.180/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/maas-controller-698c564c8b Created pod: maas-controller-698c564c8b-rgrcd 18m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-698c564c8b from 0 to 1 17m Normal Pulled pod/maas-controller-698c564c8b-rgrcd Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 429ms (429ms including waiting). Image size: 200053271 bytes. 17m Normal Created pod/maas-controller-698c564c8b-rgrcd Container created 17m Normal Started pod/maas-controller-698c564c8b-rgrcd Container started 17m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-698c564c8b-rgrcd_7a00c2a7-6fa0-41b7-b965-f5e3c4f8f9b9 became leader