=== events in opendatahub (2026-09-19T04:29:33Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 7m25s Normal Scheduled pod/odh-model-controller-7998c67dc4-rvrrj Successfully assigned opendatahub/odh-model-controller-7998c67dc4-rvrrj to ip-10-0-136-23.ec2.internal 65s Normal Scheduled pod/ai-gateway-controller-65dd9b4479-785gj Successfully assigned opendatahub/ai-gateway-controller-65dd9b4479-785gj to ip-10-0-136-23.ec2.internal 7m25s Normal Scheduled pod/model-serving-api-54d89bd5fc-s4h6g Successfully assigned opendatahub/model-serving-api-54d89bd5fc-s4h6g to ip-10-0-136-23.ec2.internal 7m13s Normal Scheduled pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Successfully assigned opendatahub/llmisvc-controller-manager-777c6bd7d5-nb4vg to ip-10-0-138-12.ec2.internal 7m55s Normal Scheduled pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Successfully assigned opendatahub/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 to ip-10-0-136-23.ec2.internal 4m25s Normal Scheduled pod/maas-controller-6f7c65cb45-brb6f Successfully assigned opendatahub/maas-controller-6f7c65cb45-brb6f to ip-10-0-138-12.ec2.internal 7m55s Normal Scheduled pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Successfully assigned opendatahub/opendatahub-operator-controller-manager-78d8fdcb89-pp77f to ip-10-0-138-12.ec2.internal 7m13s Normal Scheduled pod/kserve-controller-manager-d6f4b65b8-h2rl2 Successfully assigned opendatahub/kserve-controller-manager-d6f4b65b8-h2rl2 to ip-10-0-136-23.ec2.internal 4m25s Normal Scheduled pod/maas-controller-75dc6cbdf-ppfzc Successfully assigned opendatahub/maas-controller-75dc6cbdf-ppfzc to ip-10-0-138-12.ec2.internal 7m55s Normal Scheduled pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Successfully assigned opendatahub/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 to ip-10-0-132-168.ec2.internal 49s Normal Scheduled pod/ai-gateway-controller-cb8bb47bd-nxcj7 Successfully assigned opendatahub/ai-gateway-controller-cb8bb47bd-nxcj7 to ip-10-0-136-23.ec2.internal 7m59s Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 requirements not yet checked 7m59s Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 one or more requirements couldn't be found 7m56s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-78d8fdcb89 Created pod: opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 7m56s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-78d8fdcb89 Created pod: opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 7m56s Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-78d8fdcb89 from 0 to 3 7m56s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-78d8fdcb89 Created pod: opendatahub-operator-controller-manager-78d8fdcb89-pp77f 7m56s Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 all requirements found, attempting install 7m55s Normal AddedInterface pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Add eth0 [10.134.0.25/23] from ovn-kubernetes 7m55s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m55s Normal AddedInterface pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Add eth0 [10.133.0.19/23] from ovn-kubernetes 7m55s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m55s Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 waiting for install components to report healthy 7m55s Normal AddedInterface pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Add eth0 [10.132.0.30/23] from ovn-kubernetes 7m55s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m54s Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 7m53s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 2.901s (2.901s including waiting). Image size: 293036549 bytes. 7m53s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 2.867s (2.867s including waiting). Image size: 293036549 bytes. 7m53s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 2.842s (2.842s including waiting). Image size: 293036549 bytes. 7m52s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Created container: copy-manifests 7m52s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Created container: copy-manifests 7m52s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Started container copy-manifests 7m52s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Started container copy-manifests 7m52s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Created container: copy-manifests 7m52s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Started container copy-manifests 7m51s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m51s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m51s Normal Pulling pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m50s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 290ms (291ms including waiting). Image size: 293036549 bytes. 7m50s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 514ms (514ms including waiting). Image size: 293036549 bytes. 7m50s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Created container: manager 7m50s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Started container manager 7m50s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Created container: manager 7m50s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-cwgg9 Started container manager 7m50s Normal Created pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Created container: manager 7m50s Normal Started pod/opendatahub-operator-controller-manager-78d8fdcb89-pp77f Started container manager 7m50s Normal Pulled pod/opendatahub-operator-controller-manager-78d8fdcb89-fkdt6 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 434ms (434ms including waiting). Image size: 293036549 bytes. 7m50s Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-78d8fdcb89-cwgg9_419c2d2a-948d-4570-b794-945c1e5229a5 became leader 7m38s Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 install strategy completed with no errors 7m26s Normal SuccessfulCreate replicaset/odh-model-controller-7998c67dc4 Created pod: odh-model-controller-7998c67dc4-rvrrj 7m26s Warning FailedMount pod/odh-model-controller-7998c67dc4-rvrrj MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 7m26s Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-7998c67dc4 from 0 to 1 7m25s Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-54d89bd5fc from 0 to 1 7m25s Normal SuccessfulCreate replicaset/model-serving-api-54d89bd5fc Created pod: model-serving-api-54d89bd5fc-s4h6g 7m25s Warning FailedMount pod/model-serving-api-54d89bd5fc-s4h6g MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 7m25s Warning FailedCreate replicaset/model-serving-api-54d89bd5fc Error creating: pods "model-serving-api-54d89bd5fc-" is forbidden: error looking up service account opendatahub/model-serving-api: serviceaccount "model-serving-api" not found 7m25s Normal Pulling pod/odh-model-controller-7998c67dc4-rvrrj Pulling image "quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2" 7m25s Normal AddedInterface pod/odh-model-controller-7998c67dc4-rvrrj Add eth0 [10.132.0.32/23] from ovn-kubernetes 7m24s Normal AddedInterface pod/model-serving-api-54d89bd5fc-s4h6g Add eth0 [10.132.0.33/23] from ovn-kubernetes 7m24s Normal Pulling pod/model-serving-api-54d89bd5fc-s4h6g Pulling image "quay.io/opendatahub/odh-model-serving-api:odh-v3.5-EA2" 7m20s Normal Pulled pod/odh-model-controller-7998c67dc4-rvrrj Successfully pulled image "quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2" in 4.758s (4.758s including waiting). Image size: 238576181 bytes. 7m20s Normal Started pod/odh-model-controller-7998c67dc4-rvrrj Started container manager 7m20s Normal Created pod/odh-model-controller-7998c67dc4-rvrrj Created container: manager 7m20s Normal Pulled pod/model-serving-api-54d89bd5fc-s4h6g Successfully pulled image "quay.io/opendatahub/odh-model-serving-api:odh-v3.5-EA2" in 4.106s (4.106s including waiting). Image size: 187479094 bytes. 7m20s Normal Created pod/model-serving-api-54d89bd5fc-s4h6g Created container: server 7m20s Normal Started pod/model-serving-api-54d89bd5fc-s4h6g Started container server 7m19s Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-7998c67dc4-rvrrj_0c5ae247-6f4a-424f-b269-154868de78e3 became leader 7m14s Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-d6f4b65b8 from 0 to 1 7m14s Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-777c6bd7d5 from 0 to 1 7m13s Normal SuccessfulCreate replicaset/kserve-controller-manager-d6f4b65b8 Created pod: kserve-controller-manager-d6f4b65b8-h2rl2 7m13s Normal SuccessfulCreate replicaset/llmisvc-controller-manager-777c6bd7d5 Created pod: llmisvc-controller-manager-777c6bd7d5-nb4vg 7m13s Warning FailedMount pod/kserve-controller-manager-d6f4b65b8-h2rl2 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 7m13s Normal Pulling pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2" 7m13s Normal AddedInterface pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Add eth0 [10.133.0.23/23] from ovn-kubernetes 7m12s Normal AddedInterface pod/kserve-controller-manager-d6f4b65b8-h2rl2 Add eth0 [10.132.0.34/23] from ovn-kubernetes 7m12s Normal Pulling pod/kserve-controller-manager-d6f4b65b8-h2rl2 Pulling image "quay.io/opendatahub/kserve-controller:odh-v3.5-EA2" 7m10s Normal Created pod/kserve-controller-manager-d6f4b65b8-h2rl2 Created container: manager 7m10s Normal Pulled pod/kserve-controller-manager-d6f4b65b8-h2rl2 Successfully pulled image "quay.io/opendatahub/kserve-controller:odh-v3.5-EA2" in 2.139s (2.139s including waiting). Image size: 241618392 bytes. 7m10s Normal Started pod/kserve-controller-manager-d6f4b65b8-h2rl2 Started container manager 7m10s Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-777c6bd7d5-nb4vg_0d7a952e-2b0c-4c5e-8d58-0a95923d16ea became leader 7m10s Normal Pulled pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2" in 2.487s (2.487s including waiting). Image size: 133973437 bytes. 7m10s Normal Created pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Created container: manager 7m10s Normal Started pod/llmisvc-controller-manager-777c6bd7d5-nb4vg Started container manager 7m9s Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-d6f4b65b8-h2rl2_ec29f2ff-5e78-45a4-bb92-1adca005eef6 became leader 4m26s Normal SuccessfulCreate replicaset/maas-controller-75dc6cbdf Created pod: maas-controller-75dc6cbdf-ppfzc 4m26s Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-75dc6cbdf from 0 to 1 4m26s Normal SuccessfulCreate replicaset/maas-controller-6f7c65cb45 Created pod: maas-controller-6f7c65cb45-brb6f 4m26s Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-6f7c65cb45 from 0 to 1 4m25s Normal AddedInterface pod/maas-controller-75dc6cbdf-ppfzc Add eth0 [10.133.0.29/23] from ovn-kubernetes 4m25s Normal Pulling pod/maas-controller-75dc6cbdf-ppfzc Pulling image "quay.io/opendatahub/maas-controller:latest" 4m25s Normal Pulling pod/maas-controller-6f7c65cb45-brb6f Pulling image "quay.io/opendatahub/maas-controller:latest" 4m25s Normal AddedInterface pod/maas-controller-6f7c65cb45-brb6f Add eth0 [10.133.0.30/23] from ovn-kubernetes 4m23s Normal Pulled pod/maas-controller-6f7c65cb45-brb6f Successfully pulled image "quay.io/opendatahub/maas-controller:latest" in 2.38s (2.38s including waiting). Image size: 198037314 bytes. 4m23s Normal Pulled pod/maas-controller-75dc6cbdf-ppfzc Successfully pulled image "quay.io/opendatahub/maas-controller:latest" in 2.542s (2.542s including waiting). Image size: 198037314 bytes. 4m22s Normal Created pod/maas-controller-6f7c65cb45-brb6f Created container: manager 4m22s Normal Started pod/maas-controller-6f7c65cb45-brb6f Started container manager 4m22s Normal Created pod/maas-controller-75dc6cbdf-ppfzc Created container: manager 4m22s Normal Started pod/maas-controller-75dc6cbdf-ppfzc Started container manager 4m21s Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-75dc6cbdf-ppfzc_c2834810-e780-45d0-91e0-4e220e53decd became leader 4m11s Normal Killing pod/maas-controller-75dc6cbdf-ppfzc Stopping container manager 4m11s Normal SuccessfulDelete replicaset/maas-controller-75dc6cbdf Deleted pod: maas-controller-75dc6cbdf-ppfzc 4m11s Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-75dc6cbdf from 1 to 0 3m52s Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-6f7c65cb45-brb6f_83048bb7-8442-4c70-951a-96926f1615d6 became leader 65s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-65dd9b4479 from 0 to 1 65s Normal SuccessfulCreate replicaset/ai-gateway-controller-65dd9b4479 Created pod: ai-gateway-controller-65dd9b4479-785gj 65s Normal AddedInterface pod/ai-gateway-controller-65dd9b4479-785gj Add eth0 [10.132.0.48/23] from ovn-kubernetes 65s Normal Pulling pod/ai-gateway-controller-65dd9b4479-785gj Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" 62s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-65dd9b4479-785gj_d7e9d026-5bac-42e7-96ee-67071a40ddb3 became leader 62s Normal Pulled pod/ai-gateway-controller-65dd9b4479-785gj Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" in 2.196s (2.196s including waiting). Image size: 165741046 bytes. 62s Normal Created pod/ai-gateway-controller-65dd9b4479-785gj Created container: manager 62s Normal Started pod/ai-gateway-controller-65dd9b4479-785gj Started container manager 50s Normal SuccessfulCreate replicaset/ai-gateway-controller-cb8bb47bd Created pod: ai-gateway-controller-cb8bb47bd-nxcj7 50s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-cb8bb47bd from 0 to 1 49s Normal Pulling pod/ai-gateway-controller-cb8bb47bd-nxcj7 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" 49s Normal AddedInterface pod/ai-gateway-controller-cb8bb47bd-nxcj7 Add eth0 [10.132.0.49/23] from ovn-kubernetes 49s Normal Pulled pod/ai-gateway-controller-cb8bb47bd-nxcj7 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" in 368ms (368ms including waiting). Image size: 165741046 bytes. 49s Normal Created pod/ai-gateway-controller-cb8bb47bd-nxcj7 Created container: manager 49s Normal Started pod/ai-gateway-controller-cb8bb47bd-nxcj7 Started container manager 37s Normal SuccessfulDelete replicaset/ai-gateway-controller-65dd9b4479 Deleted pod: ai-gateway-controller-65dd9b4479-785gj 37s Normal Killing pod/ai-gateway-controller-65dd9b4479-785gj Stopping container manager 37s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-65dd9b4479 from 1 to 0 18s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-cb8bb47bd-nxcj7_f0f538c2-4ec8-48ae-a652-c9e4db1e823b became leader