=== events in opendatahub (2026-11-03T22:10:27Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 29m Normal Scheduled pod/ai-gateway-controller-76cf67fc7b-5hk42 Successfully assigned opendatahub/ai-gateway-controller-76cf67fc7b-5hk42 to ip-10-0-137-212.ec2.internal 33m Normal Scheduled pod/maas-controller-579cf7c6d-smhz7 Successfully assigned opendatahub/maas-controller-579cf7c6d-smhz7 to ip-10-0-131-253.ec2.internal 32m Normal Scheduled pod/odh-model-controller-769875db56-2cctc Successfully assigned opendatahub/odh-model-controller-769875db56-2cctc to ip-10-0-131-253.ec2.internal 32m Normal Scheduled pod/model-serving-api-75bd89fdfc-vppdx Successfully assigned opendatahub/model-serving-api-75bd89fdfc-vppdx to ip-10-0-131-253.ec2.internal 32m Normal Scheduled pod/kserve-controller-manager-776cc66d47-lrr9z Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-lrr9z to ip-10-0-131-253.ec2.internal 33m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr to ip-10-0-131-253.ec2.internal 21m Normal Scheduled pod/maas-controller-6d575fdb56-mtkvs Successfully assigned opendatahub/maas-controller-6d575fdb56-mtkvs to ip-10-0-131-253.ec2.internal 30m Normal Scheduled pod/ai-gateway-controller-86c7698f44-8wtv7 Successfully assigned opendatahub/ai-gateway-controller-86c7698f44-8wtv7 to ip-10-0-137-212.ec2.internal 20m Normal Scheduled pod/maas-controller-6d575fdb56-jwwm2 Successfully assigned opendatahub/maas-controller-6d575fdb56-jwwm2 to ip-10-0-131-253.ec2.internal 31m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-xd5q8 Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-xd5q8 to ip-10-0-142-174.ec2.internal 33m Normal Scheduled pod/maas-controller-6d575fdb56-8br4h Successfully assigned opendatahub/maas-controller-6d575fdb56-8br4h to ip-10-0-131-253.ec2.internal 32m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-mr24m Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-mr24m to ip-10-0-142-174.ec2.internal 33m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 to ip-10-0-131-253.ec2.internal 33m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 33m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-kvrbr 33m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Add eth0 [10.132.0.47/23] from ovn-kubernetes 33m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 33m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Container created 33m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Container started 33m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-kvrbr Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.909s (2.909s including waiting). Image size: 195729000 bytes. 33m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-kvrbr_98725eb7-7bc4-47ae-9291-583b1997aca9 became leader 33m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 33m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-smhz7 33m Normal SuccessfulCreate replicaset/maas-controller-6d575fdb56 Created pod: maas-controller-6d575fdb56-8br4h 33m Normal AddedInterface pod/maas-controller-6d575fdb56-8br4h Add eth0 [10.132.0.50/23] from ovn-kubernetes 33m Normal Pulling pod/maas-controller-579cf7c6d-smhz7 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 33m Normal AddedInterface pod/maas-controller-579cf7c6d-smhz7 Add eth0 [10.132.0.49/23] from ovn-kubernetes 33m Normal Created pod/maas-controller-579cf7c6d-smhz7 Container created 33m Normal Pulled pod/maas-controller-579cf7c6d-smhz7 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.869s (2.869s including waiting). Image size: 200053271 bytes. 33m Normal Pulled pod/maas-controller-6d575fdb56-8br4h Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.691s (2.691s including waiting). Image size: 200053271 bytes. 33m Normal Started pod/maas-controller-579cf7c6d-smhz7 Container started 33m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-smhz7_46b5d8a3-b58f-463a-b556-c3d6cc6ac0ad became leader 33m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 33m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-95xs7 33m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 33m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Add eth0 [10.132.0.51/23] from ovn-kubernetes 33m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 32m Normal Killing pod/maas-controller-579cf7c6d-smhz7 Stopping container manager 32m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-smhz7 32m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 32m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 11.08s (11.08s including waiting). Image size: 1089502687 bytes. 32m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Container created 32m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-95xs7 Container started 32m Normal Pulling pod/odh-model-controller-769875db56-2cctc Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 32m Normal Pulling pod/kserve-controller-manager-776cc66d47-lrr9z Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 32m Normal AddedInterface pod/odh-model-controller-769875db56-2cctc Add eth0 [10.132.0.54/23] from ovn-kubernetes 32m Normal AddedInterface pod/model-serving-api-75bd89fdfc-vppdx Add eth0 [10.132.0.53/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-vppdx 32m Normal Pulling pod/model-serving-api-75bd89fdfc-vppdx Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 32m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-mr24m Add eth0 [10.134.0.34/23] from ovn-kubernetes 32m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-mr24m Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 32m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 32m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-lrr9z 32m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 32m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-lrr9z Add eth0 [10.132.0.52/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-mr24m 32m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 32m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 32m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-2cctc 32m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-mr24m Container started 32m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-mr24m Container created 32m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-mr24m Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.023s (2.023s including waiting). Image size: 235100981 bytes. 32m Normal Started pod/model-serving-api-75bd89fdfc-vppdx Container started 32m Normal Created pod/model-serving-api-75bd89fdfc-vppdx Container created 32m Normal Pulled pod/model-serving-api-75bd89fdfc-vppdx Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.171s (2.171s including waiting). Image size: 184925761 bytes. 32m Normal Pulled pod/kserve-controller-manager-776cc66d47-lrr9z Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.809s (2.809s including waiting). Image size: 247676581 bytes. 32m Normal Created pod/kserve-controller-manager-776cc66d47-lrr9z Container created 32m Normal Started pod/kserve-controller-manager-776cc66d47-lrr9z Container started 32m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-2cctc_ae3850ac-5d59-4ed7-9b7e-747f953e89af became leader 32m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-lrr9z_a3ecab54-fc97-43dd-97e7-947f1e41204b became leader 32m Normal Started pod/odh-model-controller-769875db56-2cctc Container started 32m Normal Created pod/odh-model-controller-769875db56-2cctc Container created 32m Normal Pulled pod/odh-model-controller-769875db56-2cctc Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.115s (3.115s including waiting). Image size: 236755567 bytes. 32m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-mr24m_a536eff5-b08c-41ef-9805-8b6476ba07bd became leader 32m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6d575fdb56-8br4h_b20f8656-68d2-4443-ab12-206e0c48a627 became leader 31m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-xd5q8 31m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 31m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-xd5q8 Add eth0 [10.134.0.37/23] from ovn-kubernetes 31m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-xd5q8 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 31m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-xd5q8 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 278ms (278ms including waiting). Image size: 235100981 bytes. 31m Normal Created pod/llmisvc-controller-manager-6466c554d8-xd5q8 Container created 31m Normal Started pod/llmisvc-controller-manager-6466c554d8-xd5q8 Container started 31m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 31m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-mr24m 31m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-mr24m Stopping container manager 31m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-xd5q8_5a359726-3a29-446c-8f5d-a43b1920b450 became leader 30m Normal Pulled pod/maas-controller-6d575fdb56-8br4h Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 414ms (414ms including waiting). Image size: 200053271 bytes. 30m Normal Created pod/maas-controller-6d575fdb56-8br4h Container created 30m Warning Unhealthy pod/maas-controller-6d575fdb56-8br4h Readiness probe failed: Get "http://10.132.0.50:8081/readyz": dial tcp 10.132.0.50:8081: connect: connection refused 30m Normal Pulling pod/maas-controller-6d575fdb56-8br4h Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 30m Normal Started pod/maas-controller-6d575fdb56-8br4h Container started 30m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6d575fdb56-8br4h_533ce707-e16e-4963-8381-ceac008892ce became leader 30m Normal SuccessfulCreate replicaset/ai-gateway-controller-86c7698f44 Created pod: ai-gateway-controller-86c7698f44-8wtv7 30m Normal Pulling pod/ai-gateway-controller-86c7698f44-8wtv7 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-211" 30m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-86c7698f44 from 0 to 1 30m Normal AddedInterface pod/ai-gateway-controller-86c7698f44-8wtv7 Add eth0 [10.133.0.34/23] from ovn-kubernetes 30m Normal Created pod/ai-gateway-controller-86c7698f44-8wtv7 Container created 30m Normal Pulled pod/ai-gateway-controller-86c7698f44-8wtv7 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-211" in 4.163s (4.163s including waiting). Image size: 184147363 bytes. 30m Normal Started pod/ai-gateway-controller-86c7698f44-8wtv7 Container started 30m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-86c7698f44-8wtv7_0210441d-5766-4b66-85fd-0d06738a8302 became leader 29m Normal Created pod/ai-gateway-controller-76cf67fc7b-5hk42 Container created 29m Normal Started pod/ai-gateway-controller-76cf67fc7b-5hk42 Container started 29m Normal Pulling pod/ai-gateway-controller-76cf67fc7b-5hk42 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-211" 29m Normal AddedInterface pod/ai-gateway-controller-76cf67fc7b-5hk42 Add eth0 [10.133.0.36/23] from ovn-kubernetes 29m Normal Pulled pod/ai-gateway-controller-76cf67fc7b-5hk42 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-211" in 280ms (280ms including waiting). Image size: 184147363 bytes. 29m Normal SuccessfulCreate replicaset/ai-gateway-controller-76cf67fc7b Created pod: ai-gateway-controller-76cf67fc7b-5hk42 29m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-76cf67fc7b from 0 to 1 29m Normal SuccessfulDelete replicaset/ai-gateway-controller-86c7698f44 Deleted pod: ai-gateway-controller-86c7698f44-8wtv7 29m Normal Killing pod/ai-gateway-controller-86c7698f44-8wtv7 Stopping container manager 29m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-86c7698f44 from 1 to 0 29m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-76cf67fc7b-5hk42_ce3b771a-772d-4bab-9735-2d583edbae89 became leader 21m Normal Killing pod/maas-controller-6d575fdb56-8br4h Stopping container manager 21m Normal SuccessfulDelete replicaset/maas-controller-6d575fdb56 Deleted pod: maas-controller-6d575fdb56-8br4h 21m Normal SuccessfulCreate replicaset/maas-controller-6d575fdb56 Created pod: maas-controller-6d575fdb56-mtkvs 21m Normal Started pod/maas-controller-6d575fdb56-mtkvs Container started 21m Normal AddedInterface pod/maas-controller-6d575fdb56-mtkvs Add eth0 [10.132.0.157/23] from ovn-kubernetes 21m Normal Pulling pod/maas-controller-6d575fdb56-mtkvs Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 21m Normal Pulled pod/maas-controller-6d575fdb56-mtkvs Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 444ms (444ms including waiting). Image size: 200053271 bytes. 21m Normal Created pod/maas-controller-6d575fdb56-mtkvs Container created 20m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6d575fdb56-mtkvs_7f8b5a2c-1842-4af0-bbc5-505c4620396e became leader 20m Normal Killing pod/maas-controller-6d575fdb56-mtkvs Stopping container manager 20m Normal SuccessfulDelete replicaset/maas-controller-6d575fdb56 Deleted pod: maas-controller-6d575fdb56-mtkvs 20m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-6d575fdb56 from 1 to 0 20m Normal SuccessfulCreate replicaset/maas-controller-6d575fdb56 Created pod: maas-controller-6d575fdb56-jwwm2 20m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-6d575fdb56 from 0 to 1 20m Normal Started pod/maas-controller-6d575fdb56-jwwm2 Container started 20m Normal Created pod/maas-controller-6d575fdb56-jwwm2 Container created 20m Normal Pulled pod/maas-controller-6d575fdb56-jwwm2 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 321ms (321ms including waiting). Image size: 200053271 bytes. 20m Normal Pulling pod/maas-controller-6d575fdb56-jwwm2 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 20m Normal AddedInterface pod/maas-controller-6d575fdb56-jwwm2 Add eth0 [10.132.0.160/23] from ovn-kubernetes 20m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6d575fdb56-jwwm2_fc052155-7b5e-48f4-babd-46b01d44ac99 became leader