=== events in opendatahub (2026-09-19T10:03:57Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 65s Normal Scheduled pod/ai-gateway-controller-65dd9b4479-twn9b Successfully assigned opendatahub/ai-gateway-controller-65dd9b4479-twn9b to ip-10-0-142-110.ec2.internal 6m52s Normal Scheduled pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Successfully assigned opendatahub/llmisvc-controller-manager-777c6bd7d5-kfhgh to ip-10-0-142-110.ec2.internal 7m1s Normal Scheduled pod/model-serving-api-54d89bd5fc-gc5ld Successfully assigned opendatahub/model-serving-api-54d89bd5fc-gc5ld to ip-10-0-142-110.ec2.internal 7m1s Normal Scheduled pod/odh-model-controller-7998c67dc4-xc4hb Successfully assigned opendatahub/odh-model-controller-7998c67dc4-xc4hb to ip-10-0-143-218.ec2.internal 3m41s Normal Scheduled pod/maas-controller-7f54d4b44b-9h6v2 Successfully assigned opendatahub/maas-controller-7f54d4b44b-9h6v2 to ip-10-0-143-218.ec2.internal 7m33s Normal Scheduled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Successfully assigned opendatahub/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b to ip-10-0-143-218.ec2.internal 7m33s Normal Scheduled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Successfully assigned opendatahub/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl to ip-10-0-142-110.ec2.internal 50s Normal Scheduled pod/ai-gateway-controller-5fbc954fb5-s87tn Successfully assigned opendatahub/ai-gateway-controller-5fbc954fb5-s87tn to ip-10-0-142-110.ec2.internal 6m52s Normal Scheduled pod/kserve-controller-manager-d6f4b65b8-lws9t Successfully assigned opendatahub/kserve-controller-manager-d6f4b65b8-lws9t to ip-10-0-142-110.ec2.internal 3m41s Normal Scheduled pod/maas-controller-75dc6cbdf-82jnc Successfully assigned opendatahub/maas-controller-75dc6cbdf-82jnc to ip-10-0-143-218.ec2.internal 7m33s Normal Scheduled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Successfully assigned opendatahub/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm to ip-10-0-139-28.ec2.internal 7m37s Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 one or more requirements couldn't be found 7m37s Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 requirements not yet checked 7m34s Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-5c8c4cc7bf from 0 to 3 7m34s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-5c8c4cc7bf Created pod: opendatahub-operator-controller-manager-5c8c4cc7bf-q692b 7m34s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-5c8c4cc7bf Created pod: opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl 7m34s Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-5c8c4cc7bf Created pod: opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm 7m34s Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 all requirements found, attempting install 7m33s Normal AddedInterface pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Add eth0 [10.132.0.34/23] from ovn-kubernetes 7m33s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m33s Normal AddedInterface pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Add eth0 [10.134.0.14/23] from ovn-kubernetes 7m33s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m33s Normal AddedInterface pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Add eth0 [10.133.0.24/23] from ovn-kubernetes 7m33s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m30s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Created container: copy-manifests 7m30s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 2.841s (2.841s including waiting). Image size: 293036549 bytes. 7m30s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Started container copy-manifests 7m30s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Started container copy-manifests 7m30s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Created container: copy-manifests 7m30s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 2.952s (2.952s including waiting). Image size: 293036549 bytes. 7m29s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m29s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Started container copy-manifests 7m29s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Created container: copy-manifests 7m29s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 4.07s (4.07s including waiting). Image size: 293036549 bytes. 7m28s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Created container: manager 7m28s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m28s Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl_eded3fb5-7e16-4b6f-973b-3f1ceacc1c4d became leader 7m28s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Started container manager 7m28s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-z5jjl Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 338ms (338ms including waiting). Image size: 293036549 bytes. 7m27s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Created container: manager 7m27s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 308ms (308ms including waiting). Image size: 293036549 bytes. 7m27s Normal Pulling pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Pulling image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" 7m27s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Started container manager 7m27s Normal Started pod/opendatahub-operator-controller-manager-5c8c4cc7bf-q692b Started container manager 7m27s Normal Created pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Created container: manager 7m27s Normal Pulled pod/opendatahub-operator-controller-manager-5c8c4cc7bf-x7drm Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2" in 278ms (278ms including waiting). Image size: 293036549 bytes. 7m24s Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 Webhook install failed: conversionWebhook not ready 7m23s Warning AppliedWithWarnings installplan/install-n4lv4 1 warning(s) generated during installation of operator "opendatahub-operator.v3.5.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 7m23s Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 waiting for install components to report healthy 7m22s Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 7m15s Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.5.0-ea.2 install strategy completed with no errors 7m2s Warning FailedMount pod/odh-model-controller-7998c67dc4-xc4hb MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 7m2s Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-7998c67dc4 from 0 to 1 7m2s Warning FailedCreate replicaset/model-serving-api-54d89bd5fc Error creating: pods "model-serving-api-54d89bd5fc-" is forbidden: error looking up service account opendatahub/model-serving-api: serviceaccount "model-serving-api" not found 7m2s Normal SuccessfulCreate replicaset/model-serving-api-54d89bd5fc Created pod: model-serving-api-54d89bd5fc-gc5ld 7m2s Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-54d89bd5fc from 0 to 1 7m2s Normal SuccessfulCreate replicaset/odh-model-controller-7998c67dc4 Created pod: odh-model-controller-7998c67dc4-xc4hb 7m1s Normal Pulling pod/odh-model-controller-7998c67dc4-xc4hb Pulling image "quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2" 7m1s Normal AddedInterface pod/odh-model-controller-7998c67dc4-xc4hb Add eth0 [10.132.0.37/23] from ovn-kubernetes 7m1s Warning FailedMount pod/model-serving-api-54d89bd5fc-gc5ld MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 7m Normal AddedInterface pod/model-serving-api-54d89bd5fc-gc5ld Add eth0 [10.134.0.17/23] from ovn-kubernetes 7m Normal Pulling pod/model-serving-api-54d89bd5fc-gc5ld Pulling image "quay.io/opendatahub/odh-model-serving-api:odh-v3.5-EA2" 6m59s Normal Started pod/model-serving-api-54d89bd5fc-gc5ld Started container server 6m59s Normal Pulled pod/model-serving-api-54d89bd5fc-gc5ld Successfully pulled image "quay.io/opendatahub/odh-model-serving-api:odh-v3.5-EA2" in 1.337s (1.337s including waiting). Image size: 187479094 bytes. 6m59s Normal Created pod/model-serving-api-54d89bd5fc-gc5ld Created container: server 6m58s Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-7998c67dc4-xc4hb_7f36e24e-488b-4c5a-8df3-1b909c91801c became leader 6m58s Normal Pulled pod/odh-model-controller-7998c67dc4-xc4hb Successfully pulled image "quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2" in 2.733s (2.733s including waiting). Image size: 238576181 bytes. 6m58s Normal Created pod/odh-model-controller-7998c67dc4-xc4hb Created container: manager 6m58s Normal Started pod/odh-model-controller-7998c67dc4-xc4hb Started container manager 6m53s Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-d6f4b65b8 from 0 to 1 6m53s Normal SuccessfulCreate replicaset/llmisvc-controller-manager-777c6bd7d5 Created pod: llmisvc-controller-manager-777c6bd7d5-kfhgh 6m53s Warning FailedMount pod/kserve-controller-manager-d6f4b65b8-lws9t MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 6m53s Normal SuccessfulCreate replicaset/kserve-controller-manager-d6f4b65b8 Created pod: kserve-controller-manager-d6f4b65b8-lws9t 6m53s Normal AddedInterface pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Add eth0 [10.134.0.19/23] from ovn-kubernetes 6m53s Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-777c6bd7d5 from 0 to 1 6m52s Normal AddedInterface pod/kserve-controller-manager-d6f4b65b8-lws9t Add eth0 [10.134.0.18/23] from ovn-kubernetes 6m52s Normal Pulling pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2" 6m52s Normal Pulling pod/kserve-controller-manager-d6f4b65b8-lws9t Pulling image "quay.io/opendatahub/kserve-controller:odh-v3.5-EA2" 6m49s Normal Created pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Created container: manager 6m49s Normal Created pod/kserve-controller-manager-d6f4b65b8-lws9t Created container: manager 6m49s Normal Pulled pod/kserve-controller-manager-d6f4b65b8-lws9t Successfully pulled image "quay.io/opendatahub/kserve-controller:odh-v3.5-EA2" in 2.465s (2.465s including waiting). Image size: 241618392 bytes. 6m49s Normal Started pod/kserve-controller-manager-d6f4b65b8-lws9t Started container manager 6m49s Normal Started pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Started container manager 6m49s Normal Pulled pod/llmisvc-controller-manager-777c6bd7d5-kfhgh Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2" in 3.059s (3.059s including waiting). Image size: 133973437 bytes. 6m49s Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-d6f4b65b8-lws9t_f7dce8b6-3bbc-48e8-8319-2c135b39393d became leader 6m49s Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-777c6bd7d5-kfhgh_a724e42f-733a-44f1-8f18-bdc4c7c5a4d2 became leader 3m42s Normal SuccessfulCreate replicaset/maas-controller-75dc6cbdf Created pod: maas-controller-75dc6cbdf-82jnc 3m42s Normal SuccessfulCreate replicaset/maas-controller-7f54d4b44b Created pod: maas-controller-7f54d4b44b-9h6v2 3m42s Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-75dc6cbdf from 0 to 1 3m42s Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-7f54d4b44b from 0 to 1 3m41s Normal Pulling pod/maas-controller-75dc6cbdf-82jnc Pulling image "quay.io/opendatahub/maas-controller:latest" 3m41s Normal AddedInterface pod/maas-controller-75dc6cbdf-82jnc Add eth0 [10.132.0.43/23] from ovn-kubernetes 3m41s Normal AddedInterface pod/maas-controller-7f54d4b44b-9h6v2 Add eth0 [10.132.0.44/23] from ovn-kubernetes 3m41s Normal Pulling pod/maas-controller-7f54d4b44b-9h6v2 Pulling image "quay.io/opendatahub/maas-controller:latest" 3m39s Normal Started pod/maas-controller-75dc6cbdf-82jnc Started container manager 3m39s Normal Created pod/maas-controller-75dc6cbdf-82jnc Created container: manager 3m39s Normal Pulled pod/maas-controller-7f54d4b44b-9h6v2 Successfully pulled image "quay.io/opendatahub/maas-controller:latest" in 2.514s (2.514s including waiting). Image size: 198037314 bytes. 3m39s Normal Created pod/maas-controller-7f54d4b44b-9h6v2 Created container: manager 3m39s Normal Started pod/maas-controller-7f54d4b44b-9h6v2 Started container manager 3m39s Normal Pulled pod/maas-controller-75dc6cbdf-82jnc Successfully pulled image "quay.io/opendatahub/maas-controller:latest" in 2.665s (2.665s including waiting). Image size: 198037314 bytes. 3m38s Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-75dc6cbdf-82jnc_3eb8a1eb-df02-4739-bed8-cdb4244e0689 became leader 3m27s Normal Killing pod/maas-controller-75dc6cbdf-82jnc Stopping container manager 3m27s Normal SuccessfulDelete replicaset/maas-controller-75dc6cbdf Deleted pod: maas-controller-75dc6cbdf-82jnc 3m27s Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-75dc6cbdf from 1 to 0 3m10s Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-7f54d4b44b-9h6v2_2be472a7-ad0d-4c16-9b80-cf0abec8034e became leader 66s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-65dd9b4479 from 0 to 1 66s Normal SuccessfulCreate replicaset/ai-gateway-controller-65dd9b4479 Created pod: ai-gateway-controller-65dd9b4479-twn9b 65s Normal Pulling pod/ai-gateway-controller-65dd9b4479-twn9b Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" 65s Normal AddedInterface pod/ai-gateway-controller-65dd9b4479-twn9b Add eth0 [10.134.0.36/23] from ovn-kubernetes 62s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-65dd9b4479-twn9b_427765b5-898e-46bd-be47-1d79efa3d798 became leader 62s Normal Started pod/ai-gateway-controller-65dd9b4479-twn9b Started container manager 62s Normal Created pod/ai-gateway-controller-65dd9b4479-twn9b Created container: manager 62s Normal Pulled pod/ai-gateway-controller-65dd9b4479-twn9b Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" in 2.553s (2.553s including waiting). Image size: 165741046 bytes. 51s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5fbc954fb5 from 0 to 1 51s Normal SuccessfulCreate replicaset/ai-gateway-controller-5fbc954fb5 Created pod: ai-gateway-controller-5fbc954fb5-s87tn 50s Normal Created pod/ai-gateway-controller-5fbc954fb5-s87tn Created container: manager 50s Normal AddedInterface pod/ai-gateway-controller-5fbc954fb5-s87tn Add eth0 [10.134.0.37/23] from ovn-kubernetes 50s Normal Pulling pod/ai-gateway-controller-5fbc954fb5-s87tn Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" 50s Normal Pulled pod/ai-gateway-controller-5fbc954fb5-s87tn Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37" in 376ms (376ms including waiting). Image size: 165741046 bytes. 50s Normal Started pod/ai-gateway-controller-5fbc954fb5-s87tn Started container manager 38s Normal Killing pod/ai-gateway-controller-65dd9b4479-twn9b Stopping container manager 38s Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-65dd9b4479 from 1 to 0 38s Normal SuccessfulDelete replicaset/ai-gateway-controller-65dd9b4479 Deleted pod: ai-gateway-controller-65dd9b4479-twn9b 17s Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5fbc954fb5-s87tn_d97df0f3-bc2f-4458-a032-f8d5f3d0a495 became leader