=== events in opendatahub (2026-11-02T16:11:45Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 61m Normal Scheduled pod/ai-gateway-controller-5d46d8f988-tnm4n Successfully assigned opendatahub/ai-gateway-controller-5d46d8f988-tnm4n to ip-10-0-135-181.us-west-2.compute.internal 64m Normal Scheduled pod/model-serving-api-75bd89fdfc-cn7hn Successfully assigned opendatahub/model-serving-api-75bd89fdfc-cn7hn to ip-10-0-142-141.us-west-2.compute.internal 49m Normal Scheduled pod/test-curl-6585-4b5b3c Successfully assigned opendatahub/test-curl-6585-4b5b3c to ip-10-0-130-229.us-west-2.compute.internal 49m Normal Scheduled pod/test-curl-6585-4b4e78 Successfully assigned opendatahub/test-curl-6585-4b4e78 to ip-10-0-130-229.us-west-2.compute.internal 39m Normal Scheduled pod/test-curl-6582-b983a5 Successfully assigned opendatahub/test-curl-6582-b983a5 to ip-10-0-130-229.us-west-2.compute.internal 64m Normal Scheduled pod/odh-model-controller-769875db56-bs4qx Successfully assigned opendatahub/odh-model-controller-769875db56-bs4qx to ip-10-0-135-181.us-west-2.compute.internal 21m Normal Scheduled pod/maas-controller-754d6b6455-qdgdv Successfully assigned opendatahub/maas-controller-754d6b6455-qdgdv to ip-10-0-135-181.us-west-2.compute.internal 63m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-748vc Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-748vc to ip-10-0-130-229.us-west-2.compute.internal 65m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc to ip-10-0-142-141.us-west-2.compute.internal 61m Normal Scheduled pod/ai-gateway-controller-84664687bf-sqhrz Successfully assigned opendatahub/ai-gateway-controller-84664687bf-sqhrz to ip-10-0-135-181.us-west-2.compute.internal 64m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r to ip-10-0-130-229.us-west-2.compute.internal 21m Normal Scheduled pod/maas-controller-754d6b6455-xr57v Successfully assigned opendatahub/maas-controller-754d6b6455-xr57v to ip-10-0-135-181.us-west-2.compute.internal 64m Normal Scheduled pod/kserve-controller-manager-776cc66d47-pmdck Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-pmdck to ip-10-0-142-141.us-west-2.compute.internal 64m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd to ip-10-0-130-229.us-west-2.compute.internal 64m Normal Scheduled pod/maas-controller-579cf7c6d-kdt5p Successfully assigned opendatahub/maas-controller-579cf7c6d-kdt5p to ip-10-0-135-181.us-west-2.compute.internal 64m Normal Scheduled pod/maas-controller-754d6b6455-5rk2q Successfully assigned opendatahub/maas-controller-754d6b6455-5rk2q to ip-10-0-135-181.us-west-2.compute.internal 65m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 65m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-h9cpc 65m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 65m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Add eth0 [10.134.0.31/23] from ovn-kubernetes 65m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Container started 65m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.065s (2.065s including waiting). Image size: 195729000 bytes. 65m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-h9cpc Container created 65m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-h9cpc_07e4fe63-5acf-4063-94f9-fd2e9b2d2ca5 became leader 64m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 64m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-kdt5p 64m Normal AddedInterface pod/maas-controller-579cf7c6d-kdt5p Add eth0 [10.133.0.22/23] from ovn-kubernetes 64m Normal Pulling pod/maas-controller-579cf7c6d-kdt5p Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 64m Normal AddedInterface pod/maas-controller-754d6b6455-5rk2q Add eth0 [10.133.0.23/23] from ovn-kubernetes 64m Normal Pulling pod/maas-controller-754d6b6455-5rk2q Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 64m Normal SuccessfulCreate replicaset/maas-controller-754d6b6455 Created pod: maas-controller-754d6b6455-5rk2q 64m Normal Pulled pod/maas-controller-754d6b6455-5rk2q Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.393s (3.393s including waiting). Image size: 200053271 bytes. 64m Normal Started pod/maas-controller-754d6b6455-5rk2q Container started 64m Normal Created pod/maas-controller-754d6b6455-5rk2q Container created 64m Normal Started pod/maas-controller-579cf7c6d-kdt5p Container started 64m Normal Created pod/maas-controller-579cf7c6d-kdt5p Container created 64m Normal Pulled pod/maas-controller-579cf7c6d-kdt5p Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.758s (3.758s including waiting). Image size: 200053271 bytes. 64m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-754d6b6455-5rk2q_c5d009f7-9b5d-4455-891f-4ee7f7d96071 became leader 64m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 64m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 64m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r MountVolume.SetUp failed for volume "proxy-source" : configmap "usage-tenancy-proxy-source" not found 64m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-xqw9r 64m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Add eth0 [10.132.0.47/23] from ovn-kubernetes 64m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 64m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-bs4qx 64m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 64m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 64m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 64m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-cn7hn 64m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 64m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-9hzkd 64m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-pmdck 64m Normal AddedInterface pod/model-serving-api-75bd89fdfc-cn7hn Add eth0 [10.134.0.35/23] from ovn-kubernetes 64m Normal Pulling pod/kserve-controller-manager-776cc66d47-pmdck Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 64m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 64m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Add eth0 [10.132.0.48/23] from ovn-kubernetes 64m Normal Pulling pod/model-serving-api-75bd89fdfc-cn7hn Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 64m Normal AddedInterface pod/odh-model-controller-769875db56-bs4qx Add eth0 [10.133.0.25/23] from ovn-kubernetes 64m Normal Pulling pod/odh-model-controller-769875db56-bs4qx Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 64m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-pmdck Add eth0 [10.134.0.34/23] from ovn-kubernetes 64m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 64m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-kdt5p 64m Normal Killing pod/maas-controller-579cf7c6d-kdt5p Stopping container manager 64m Normal Pulled pod/model-serving-api-75bd89fdfc-cn7hn Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.503s (2.503s including waiting). Image size: 184925761 bytes. 64m Normal Pulled pod/kserve-controller-manager-776cc66d47-pmdck Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.726s (2.726s including waiting). Image size: 247676581 bytes. 64m Normal Started pod/odh-model-controller-769875db56-bs4qx Container started 64m Normal Pulled pod/odh-model-controller-769875db56-bs4qx Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.213s (3.213s including waiting). Image size: 236755567 bytes. 64m Normal Created pod/odh-model-controller-769875db56-bs4qx Container created 64m Normal Started pod/model-serving-api-75bd89fdfc-cn7hn Container started 64m Normal Created pod/model-serving-api-75bd89fdfc-cn7hn Container created 64m Normal Created pod/kserve-controller-manager-776cc66d47-pmdck Container created 64m Normal Started pod/kserve-controller-manager-776cc66d47-pmdck Container started 64m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Container started 64m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Container created 64m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 4.112s (4.112s including waiting). Image size: 235100981 bytes. 64m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-bs4qx_3599b695-0a99-469b-b358-dd485e7299ed became leader 64m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-pmdck_9d74e5c8-ce9f-4886-9308-7a9d157d6f39 became leader 64m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-9hzkd_5dbd2928-b7ad-4bc7-a6df-556896836fb0 became leader 64m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Container created 64m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Container started 64m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 13.834s (13.834s including waiting). Image size: 1089502687 bytes. 63m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-748vc Add eth0 [10.132.0.49/23] from ovn-kubernetes 63m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-748vc Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 63m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 63m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-748vc 63m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-748vc Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 845ms (845ms including waiting). Image size: 235100981 bytes. 63m Normal Created pod/llmisvc-controller-manager-6466c554d8-748vc Container created 63m Normal Started pod/llmisvc-controller-manager-6466c554d8-748vc Container started 62m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 62m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-9hzkd Stopping container manager 62m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-9hzkd 62m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-748vc_5c344204-80f4-47f3-9d7c-94caa7d34bba became leader 61m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5d46d8f988 from 0 to 1 61m Normal AddedInterface pod/ai-gateway-controller-5d46d8f988-tnm4n Add eth0 [10.133.0.37/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/ai-gateway-controller-5d46d8f988 Created pod: ai-gateway-controller-5d46d8f988-tnm4n 61m Normal Pulling pod/ai-gateway-controller-5d46d8f988-tnm4n Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-152" 61m Normal Created pod/ai-gateway-controller-5d46d8f988-tnm4n Container created 61m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5d46d8f988-tnm4n_f59e85c2-7ad9-40e6-bfd9-5981e0376d77 became leader 61m Normal Pulled pod/ai-gateway-controller-5d46d8f988-tnm4n Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-152" in 6.513s (6.513s including waiting). Image size: 183979409 bytes. 61m Normal Started pod/ai-gateway-controller-5d46d8f988-tnm4n Container started 61m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-84664687bf from 0 to 1 61m Normal SuccessfulCreate replicaset/ai-gateway-controller-84664687bf Created pod: ai-gateway-controller-84664687bf-sqhrz 61m Normal Pulling pod/ai-gateway-controller-84664687bf-sqhrz Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-152" 61m Normal AddedInterface pod/ai-gateway-controller-84664687bf-sqhrz Add eth0 [10.133.0.40/23] from ovn-kubernetes 61m Normal Started pod/ai-gateway-controller-84664687bf-sqhrz Container started 61m Normal Created pod/ai-gateway-controller-84664687bf-sqhrz Container created 61m Normal Pulled pod/ai-gateway-controller-84664687bf-sqhrz Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-152" in 832ms (832ms including waiting). Image size: 183979409 bytes. 61m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5d46d8f988 from 1 to 0 61m Normal Killing pod/ai-gateway-controller-5d46d8f988-tnm4n Stopping container manager 61m Normal SuccessfulDelete replicaset/ai-gateway-controller-5d46d8f988 Deleted pod: ai-gateway-controller-5d46d8f988-tnm4n 60m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-84664687bf-sqhrz_7fb26312-f4ba-4364-9006-1640be9eba55 became leader 49m Normal Pulling pod/test-curl-6585-4b5b3c Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 49m Normal AddedInterface pod/test-curl-6585-4b5b3c Add eth0 [10.132.0.103/23] from ovn-kubernetes 49m Normal Pulled pod/test-curl-6585-4b5b3c Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.859s (2.859s including waiting). Image size: 106100986 bytes. 49m Normal Created pod/test-curl-6585-4b5b3c Container created 49m Normal Started pod/test-curl-6585-4b5b3c Container started 49m Normal Killing pod/test-curl-6585-4b5b3c Stopping container test-curl-6585-4b5b3c 49m Normal Started pod/test-curl-6585-4b4e78 Container started 49m Normal Created pod/test-curl-6585-4b4e78 Container created 49m Normal Pulled pod/test-curl-6585-4b4e78 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/test-curl-6585-4b4e78 Add eth0 [10.132.0.105/23] from ovn-kubernetes 49m Normal Killing pod/test-curl-6585-4b4e78 Stopping container test-curl-6585-4b4e78 39m Normal AddedInterface pod/test-curl-6582-b983a5 Add eth0 [10.132.0.144/23] from ovn-kubernetes 39m Normal Started pod/test-curl-6582-b983a5 Container started 39m Normal Created pod/test-curl-6582-b983a5 Container created 39m Normal Pulled pod/test-curl-6582-b983a5 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 39m Normal Killing pod/test-curl-6582-b983a5 Stopping container test-curl-6582-b983a5 22m Normal SuccessfulDelete replicaset/maas-controller-754d6b6455 Deleted pod: maas-controller-754d6b6455-5rk2q 22m Normal Killing pod/maas-controller-754d6b6455-5rk2q Stopping container manager 21m Normal AddedInterface pod/maas-controller-754d6b6455-qdgdv Add eth0 [10.133.0.155/23] from ovn-kubernetes 21m Normal Pulling pod/maas-controller-754d6b6455-qdgdv Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 21m Normal SuccessfulCreate replicaset/maas-controller-754d6b6455 Created pod: maas-controller-754d6b6455-qdgdv 21m Normal Created pod/maas-controller-754d6b6455-qdgdv Container created 21m Normal Started pod/maas-controller-754d6b6455-qdgdv Container started 21m Normal Pulled pod/maas-controller-754d6b6455-qdgdv Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 1.64s (1.64s including waiting). Image size: 200053271 bytes. 21m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-754d6b6455-qdgdv_feba5a3c-4b57-4f49-a39b-08279781ae6b became leader 21m Normal Killing pod/maas-controller-754d6b6455-qdgdv Stopping container manager 21m Normal SuccessfulDelete replicaset/maas-controller-754d6b6455 Deleted pod: maas-controller-754d6b6455-qdgdv 21m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-754d6b6455 from 1 to 0 21m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-754d6b6455 from 0 to 1 21m Normal SuccessfulCreate replicaset/maas-controller-754d6b6455 Created pod: maas-controller-754d6b6455-xr57v 21m Normal Pulling pod/maas-controller-754d6b6455-xr57v Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 21m Normal AddedInterface pod/maas-controller-754d6b6455-xr57v Add eth0 [10.133.0.158/23] from ovn-kubernetes 21m Normal Started pod/maas-controller-754d6b6455-xr57v Container started 21m Normal Created pod/maas-controller-754d6b6455-xr57v Container created 21m Normal Pulled pod/maas-controller-754d6b6455-xr57v Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 1.142s (1.142s including waiting). Image size: 200053271 bytes. 21m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-754d6b6455-xr57v_140d6558-ba6b-4415-ba36-54cdb91bd29c became leader