=== events in opendatahub (2026-10-31T18:01:53Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 29m Normal Scheduled pod/maas-controller-579cf7c6d-jspb9 Successfully assigned opendatahub/maas-controller-579cf7c6d-jspb9 to ip-10-0-139-107.ec2.internal 29m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj to ip-10-0-139-107.ec2.internal 29m Normal Scheduled pod/odh-model-controller-769875db56-r6mcs Successfully assigned opendatahub/odh-model-controller-769875db56-r6mcs to ip-10-0-139-107.ec2.internal 26m Normal Scheduled pod/ai-gateway-controller-848fd4df45-tbvzh Successfully assigned opendatahub/ai-gateway-controller-848fd4df45-tbvzh to ip-10-0-129-127.ec2.internal 18m Normal Scheduled pod/maas-controller-58b696d4f-nlh9h Successfully assigned opendatahub/maas-controller-58b696d4f-nlh9h to ip-10-0-129-127.ec2.internal 29m Normal Scheduled pod/maas-controller-58b696d4f-lt9f8 Successfully assigned opendatahub/maas-controller-58b696d4f-lt9f8 to ip-10-0-139-107.ec2.internal 18m Normal Scheduled pod/maas-controller-58b696d4f-bg2ch Successfully assigned opendatahub/maas-controller-58b696d4f-bg2ch to ip-10-0-129-127.ec2.internal 26m Normal Scheduled pod/ai-gateway-controller-697b76b867-5dmn8 Successfully assigned opendatahub/ai-gateway-controller-697b76b867-5dmn8 to ip-10-0-129-127.ec2.internal 29m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s to ip-10-0-139-107.ec2.internal 29m Normal Scheduled pod/kserve-controller-manager-776cc66d47-lprq5 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-lprq5 to ip-10-0-139-107.ec2.internal 29m Normal Scheduled pod/model-serving-api-75bd89fdfc-6bncn Successfully assigned opendatahub/model-serving-api-75bd89fdfc-6bncn to ip-10-0-139-107.ec2.internal 28m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-2t6x8 Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8 to ip-10-0-139-107.ec2.internal 30m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-mb26r Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-mb26r to ip-10-0-129-127.ec2.internal 30m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 30m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-mb26r 30m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-mb26r Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 30m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-mb26r Add eth0 [10.133.0.22/23] from ovn-kubernetes 30m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-mb26r Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.517s (1.517s including waiting). Image size: 195729000 bytes. 29m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-mb26r_f4f05e40-23d9-43bf-bc23-8c9f6a2a6b18 became leader 29m Normal Pulling pod/maas-controller-579cf7c6d-jspb9 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 29m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-jspb9 29m Normal AddedInterface pod/maas-controller-579cf7c6d-jspb9 Add eth0 [10.134.0.31/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/maas-controller-58b696d4f Created pod: maas-controller-58b696d4f-lt9f8 29m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 29m Normal AddedInterface pod/maas-controller-58b696d4f-lt9f8 Add eth0 [10.134.0.32/23] from ovn-kubernetes 29m Normal Pulled pod/maas-controller-579cf7c6d-jspb9 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.788s (2.788s including waiting). Image size: 200053271 bytes. 29m Normal Pulled pod/maas-controller-58b696d4f-lt9f8 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.704s (2.705s including waiting). Image size: 200053271 bytes. 29m Normal Created pod/maas-controller-579cf7c6d-jspb9 Container created 29m Normal Started pod/maas-controller-579cf7c6d-jspb9 Container started 29m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-jspb9_56cbee33-c296-4e8a-9411-6aeb34b08108 became leader 29m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 29m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 29m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-5955s 29m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Add eth0 [10.134.0.33/23] from ovn-kubernetes 29m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 29m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-jspb9 29m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 29m Normal Killing pod/maas-controller-579cf7c6d-jspb9 Stopping container manager 29m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.672s (10.672s including waiting). Image size: 1089502687 bytes. 29m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Container started 29m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-5955s Container created 29m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-6bncn 29m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 29m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-lbsmj 29m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 29m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 29m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 29m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-r6mcs 29m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-lprq5 29m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 29m Normal AddedInterface pod/odh-model-controller-769875db56-r6mcs Add eth0 [10.134.0.37/23] from ovn-kubernetes 29m Normal Pulling pod/model-serving-api-75bd89fdfc-6bncn Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 29m Normal AddedInterface pod/model-serving-api-75bd89fdfc-6bncn Add eth0 [10.134.0.36/23] from ovn-kubernetes 29m Normal Pulling pod/odh-model-controller-769875db56-r6mcs Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 29m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-lprq5 Add eth0 [10.134.0.34/23] from ovn-kubernetes 29m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Add eth0 [10.134.0.35/23] from ovn-kubernetes 29m Normal Created pod/model-serving-api-75bd89fdfc-6bncn Container created 29m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.137s (3.137s including waiting). Image size: 235100981 bytes. 29m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Container created 29m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Container started 29m Normal Pulled pod/kserve-controller-manager-776cc66d47-lprq5 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.937s (2.937s including waiting). Image size: 247676581 bytes. 29m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-lprq5_8bad6794-2a6e-4ed4-87e8-78fb1dec32b5 became leader 29m Normal Pulled pod/model-serving-api-75bd89fdfc-6bncn Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.678s (2.678s including waiting). Image size: 184925761 bytes. 29m Normal Started pod/model-serving-api-75bd89fdfc-6bncn Container started 29m Normal Pulled pod/odh-model-controller-769875db56-r6mcs Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.562s (3.562s including waiting). Image size: 236755567 bytes. 29m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-r6mcs_3815e54a-30bd-4e79-9fbf-d5daf2fe947a became leader 29m Normal Started pod/odh-model-controller-769875db56-r6mcs Container started 29m Normal Created pod/odh-model-controller-769875db56-r6mcs Container created 29m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-lbsmj_dce145c3-57a2-4201-b561-a6c5adc8f0ef became leader 29m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-58b696d4f-lt9f8_fedccf95-c4c1-46a4-a3fc-f68147adeddb became leader 28m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 28m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-2t6x8 Add eth0 [10.134.0.39/23] from ovn-kubernetes 28m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-2t6x8 28m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-2t6x8 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 282ms (282ms including waiting). Image size: 235100981 bytes. 27m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-lbsmj 27m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 27m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-lbsmj Stopping container manager 27m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-2t6x8_099711d1-2868-4a58-ba14-904e04f59f8f became leader 27m Normal Created pod/maas-controller-58b696d4f-lt9f8 Container created 27m Normal Pulling pod/maas-controller-58b696d4f-lt9f8 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 27m Normal Pulled pod/maas-controller-58b696d4f-lt9f8 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 341ms (341ms including waiting). Image size: 200053271 bytes. 27m Normal Started pod/maas-controller-58b696d4f-lt9f8 Container started 26m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-58b696d4f-lt9f8_78767fd8-d2b2-4364-846b-b3fec417bb11 became leader 26m Normal Pulling pod/ai-gateway-controller-697b76b867-5dmn8 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" 26m Normal SuccessfulCreate replicaset/ai-gateway-controller-697b76b867 Created pod: ai-gateway-controller-697b76b867-5dmn8 26m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-697b76b867 from 0 to 1 26m Normal AddedInterface pod/ai-gateway-controller-697b76b867-5dmn8 Add eth0 [10.133.0.31/23] from ovn-kubernetes 26m Normal Pulled pod/ai-gateway-controller-697b76b867-5dmn8 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" in 1.471s (1.471s including waiting). Image size: 183979424 bytes. 26m Normal Started pod/ai-gateway-controller-697b76b867-5dmn8 Container started 26m Normal Created pod/ai-gateway-controller-697b76b867-5dmn8 Container created 26m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-697b76b867-5dmn8_0bd8d327-1410-4940-ae4a-684ea0cda97f became leader 26m Normal SuccessfulCreate replicaset/ai-gateway-controller-848fd4df45 Created pod: ai-gateway-controller-848fd4df45-tbvzh 26m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-848fd4df45 from 0 to 1 26m Normal Pulled pod/ai-gateway-controller-848fd4df45-tbvzh Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" in 392ms (392ms including waiting). Image size: 183979424 bytes. 26m Normal Pulling pod/ai-gateway-controller-848fd4df45-tbvzh Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184" 26m Normal AddedInterface pod/ai-gateway-controller-848fd4df45-tbvzh Add eth0 [10.133.0.35/23] from ovn-kubernetes 26m Normal Created pod/ai-gateway-controller-848fd4df45-tbvzh Container created 26m Normal Started pod/ai-gateway-controller-848fd4df45-tbvzh Container started 26m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-697b76b867 from 1 to 0 26m Normal Killing pod/ai-gateway-controller-697b76b867-5dmn8 Stopping container manager 26m Normal SuccessfulDelete replicaset/ai-gateway-controller-697b76b867 Deleted pod: ai-gateway-controller-697b76b867-5dmn8 26m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-mb26r Container image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" already present on machine and can be accessed by the pod 26m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-mb26r Container started 26m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-mb26r Container created 26m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-848fd4df45-tbvzh_898c8ed3-8b91-4c32-978c-4dea89a4aa2c became leader 25m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-mb26r_c787e774-bf61-4f4e-abb3-7e1e257f014d became leader 18m Normal Killing pod/maas-controller-58b696d4f-lt9f8 Stopping container manager 18m Normal SuccessfulDelete replicaset/maas-controller-58b696d4f Deleted pod: maas-controller-58b696d4f-lt9f8 18m Normal SuccessfulCreate replicaset/maas-controller-58b696d4f Created pod: maas-controller-58b696d4f-bg2ch 18m Normal Pulling pod/maas-controller-58b696d4f-bg2ch Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal AddedInterface pod/maas-controller-58b696d4f-bg2ch Add eth0 [10.133.0.79/23] from ovn-kubernetes 18m Normal Pulled pod/maas-controller-58b696d4f-bg2ch Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 1.718s (1.718s including waiting). Image size: 200053271 bytes. 18m Normal Created pod/maas-controller-58b696d4f-bg2ch Container created 18m Normal Started pod/maas-controller-58b696d4f-bg2ch Container started 18m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-58b696d4f-bg2ch_b6c70ad8-734a-415c-a071-e00c8964ed13 became leader 18m Normal Killing pod/maas-controller-58b696d4f-bg2ch Stopping container manager 18m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-58b696d4f from 1 to 0 18m Normal SuccessfulDelete replicaset/maas-controller-58b696d4f Deleted pod: maas-controller-58b696d4f-bg2ch 18m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-58b696d4f from 0 to 1 18m Normal SuccessfulCreate replicaset/maas-controller-58b696d4f Created pod: maas-controller-58b696d4f-nlh9h 18m Normal Started pod/maas-controller-58b696d4f-nlh9h Container started 18m Normal Pulling pod/maas-controller-58b696d4f-nlh9h Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal AddedInterface pod/maas-controller-58b696d4f-nlh9h Add eth0 [10.133.0.80/23] from ovn-kubernetes 18m Normal Pulled pod/maas-controller-58b696d4f-nlh9h Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 350ms (350ms including waiting). Image size: 200053271 bytes. 18m Normal Created pod/maas-controller-58b696d4f-nlh9h Container created 17m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-58b696d4f-nlh9h_cb68e6ed-98f8-4ac7-b949-b5263f89f8f7 became leader 3m11s Normal Pulling pod/llmisvc-controller-manager-6466c554d8-2t6x8 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 3m10s Normal Started pod/llmisvc-controller-manager-6466c554d8-2t6x8 Container started 3m10s Normal Created pod/llmisvc-controller-manager-6466c554d8-2t6x8 Container created 3m10s Normal Pulled pod/llmisvc-controller-manager-6466c554d8-2t6x8 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 348ms (348ms including waiting). Image size: 235100981 bytes. 2m53s Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-2t6x8_30a7b32b-e9ce-4a24-85d4-6ca8d7c5fcf3 became leader 2m23s Normal Pulled pod/kserve-controller-manager-776cc66d47-lprq5 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 320ms (320ms including waiting). Image size: 247676581 bytes. 2m23s Normal Started pod/kserve-controller-manager-776cc66d47-lprq5 Container started 2m23s Normal Created pod/kserve-controller-manager-776cc66d47-lprq5 Container created 2m23s Normal Pulling pod/kserve-controller-manager-776cc66d47-lprq5 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 2m5s Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-lprq5_1f576125-6ef8-411a-8178-be6a3693f318 became leader