=== events in opendatahub (2026-10-30T17:56:00Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 50m Normal Scheduled pod/test-curl-8644-cc17f3 Successfully assigned opendatahub/test-curl-8644-cc17f3 to ip-10-0-128-218.ec2.internal 60m Normal Scheduled pod/maas-controller-577f596567-qg4fp Successfully assigned opendatahub/maas-controller-577f596567-qg4fp to ip-10-0-135-153.ec2.internal 60m Normal Scheduled pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Successfully assigned opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs to ip-10-0-135-153.ec2.internal 43m Normal Scheduled pod/test-curl-8659-bb37db Successfully assigned opendatahub/test-curl-8659-bb37db to ip-10-0-135-153.ec2.internal 46m Normal Scheduled pod/test-curl-8659-a6d87b Successfully assigned opendatahub/test-curl-8659-a6d87b to ip-10-0-135-153.ec2.internal 31m Normal Scheduled pod/test-curl-8656-755369 Successfully assigned opendatahub/test-curl-8656-755369 to ip-10-0-128-218.ec2.internal 42m Normal Scheduled pod/test-curl-8653-c4fb43 Successfully assigned opendatahub/test-curl-8653-c4fb43 to ip-10-0-135-153.ec2.internal 42m Normal Scheduled pod/test-curl-8653-868444 Successfully assigned opendatahub/test-curl-8653-868444 to ip-10-0-135-153.ec2.internal 58m Normal Scheduled pod/ai-gateway-controller-66dd76bf94-lsdd7 Successfully assigned opendatahub/ai-gateway-controller-66dd76bf94-lsdd7 to ip-10-0-135-153.ec2.internal 42m Normal Scheduled pod/test-curl-8653-0e6800 Successfully assigned opendatahub/test-curl-8653-0e6800 to ip-10-0-135-153.ec2.internal 42m Normal Scheduled pod/test-curl-8653-0aba40 Successfully assigned opendatahub/test-curl-8653-0aba40 to ip-10-0-135-153.ec2.internal 30m Normal Scheduled pod/test-curl-8650-476664 Successfully assigned opendatahub/test-curl-8650-476664 to ip-10-0-135-153.ec2.internal 30m Normal Scheduled pod/test-curl-8650-0edcad Successfully assigned opendatahub/test-curl-8650-0edcad to ip-10-0-135-153.ec2.internal 50m Normal Scheduled pod/test-curl-8644-dd7736 Successfully assigned opendatahub/test-curl-8644-dd7736 to ip-10-0-128-218.ec2.internal 57m Normal Scheduled pod/ai-gateway-controller-575996c6dd-2tngr Successfully assigned opendatahub/ai-gateway-controller-575996c6dd-2tngr to ip-10-0-135-153.ec2.internal 63m Normal Scheduled pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Successfully assigned opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v to ip-10-0-135-153.ec2.internal 64m Normal Scheduled pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Successfully assigned opendatahub/opendatahub-operator-controller-manager-68c46f97f5-bzljn to ip-10-0-128-218.ec2.internal 60m Normal Scheduled pod/maas-controller-78487d6d64-hdjk9 Successfully assigned opendatahub/maas-controller-78487d6d64-hdjk9 to ip-10-0-135-153.ec2.internal 64m Normal Scheduled pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Successfully assigned opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 to ip-10-0-131-13.ec2.internal 13m Normal Scheduled pod/maas-controller-78487d6d64-k525h Successfully assigned opendatahub/maas-controller-78487d6d64-k525h to ip-10-0-135-153.ec2.internal 59m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-54btj Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-54btj to ip-10-0-131-13.ec2.internal 14m Normal Scheduled pod/maas-controller-78487d6d64-fkx6f Successfully assigned opendatahub/maas-controller-78487d6d64-fkx6f to ip-10-0-135-153.ec2.internal 63m Normal Scheduled pod/kserve-controller-manager-776cc66d47-6strf Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-6strf to ip-10-0-131-13.ec2.internal 64m Normal Scheduled pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Successfully assigned opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj to ip-10-0-135-153.ec2.internal 63m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq to ip-10-0-131-13.ec2.internal 64m Normal Scheduled pod/odh-observability-584f99c788-j2dld Successfully assigned opendatahub/odh-observability-584f99c788-j2dld to ip-10-0-131-13.ec2.internal 63m Normal Scheduled pod/odh-model-controller-f87fcf56c-zhx25 Successfully assigned opendatahub/odh-model-controller-f87fcf56c-zhx25 to ip-10-0-131-13.ec2.internal 63m Normal Scheduled pod/model-serving-api-75bd89fdfc-nt54h Successfully assigned opendatahub/model-serving-api-75bd89fdfc-nt54h to ip-10-0-131-13.ec2.internal 64m Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 requirements not yet checked 64m Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 one or more requirements couldn't be found 64m Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 all requirements found, attempting install 64m Normal AddedInterface pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Add eth0 [10.132.0.31/23] from ovn-kubernetes 64m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-68c46f97f5 Created pod: opendatahub-operator-controller-manager-68c46f97f5-bzljn 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal AddedInterface pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Add eth0 [10.133.0.22/23] from ovn-kubernetes 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal AddedInterface pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Add eth0 [10.134.0.20/23] from ovn-kubernetes 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 waiting for install components to report healthy 64m Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-68c46f97f5 from 0 to 3 64m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-68c46f97f5 Created pod: opendatahub-operator-controller-manager-68c46f97f5-wpvf2 64m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-68c46f97f5 Created pod: opendatahub-operator-controller-manager-68c46f97f5-k89lj 64m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.8s (2.8s including waiting). Image size: 273709574 bytes. 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Container started 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.874s (2.874s including waiting). Image size: 273709574 bytes. 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Container created 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Container started 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Container created 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Container created 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.82s (2.82s including waiting). Image size: 273709574 bytes. 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Container started 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 422ms (422ms including waiting). Image size: 273709574 bytes. 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Container started 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Container started 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Container created 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 363ms (363ms including waiting). Image size: 273709574 bytes. 64m Normal Pulling pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Container created 64m Normal Created pod/opendatahub-operator-controller-manager-68c46f97f5-wpvf2 Container created 64m Normal Started pod/opendatahub-operator-controller-manager-68c46f97f5-k89lj Container started 64m Normal Pulled pod/opendatahub-operator-controller-manager-68c46f97f5-bzljn Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 251ms (251ms including waiting). Image size: 273709574 bytes. 64m Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-68c46f97f5-k89lj_8de140c5-6e06-4029-b544-9424b3aafe06 became leader 64m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 install strategy completed with no errors 64m Normal SuccessfulCreate replicaset/odh-observability-584f99c788 Created pod: odh-observability-584f99c788-j2dld 64m Normal Issuing certificate/odh-observability-webhook-cert Issuing certificate as Secret does not exist 64m Normal ScalingReplicaSet deployment/odh-observability Scaled up replica set odh-observability-584f99c788 from 0 to 1 64m Warning FailedMount pod/odh-observability-584f99c788-j2dld MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 64m Normal Generated certificate/odh-observability-webhook-cert Stored new private key in temporary Secret resource "odh-observability-webhook-cert-8zds7" 64m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 64m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 64m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 64m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 64m Normal Requested certificate/odh-observability-webhook-cert Created new CertificateRequest resource "odh-observability-webhook-cert-1" 64m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 64m Normal cert-manager.io certificaterequest/odh-observability-webhook-cert-1 Certificate request has been approved by cert-manager.io 64m Normal CertificateIssued certificaterequest/odh-observability-webhook-cert-1 Certificate fetched from issuer successfully 64m Normal AddedInterface pod/odh-observability-584f99c788-j2dld Add eth0 [10.133.0.24/23] from ovn-kubernetes 64m Normal Issuing certificate/odh-observability-webhook-cert The certificate has been successfully issued 64m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 64m Normal Pulling pod/odh-observability-584f99c788-j2dld Pulling image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" 64m Normal Pulled pod/odh-observability-584f99c788-j2dld Successfully pulled image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" in 2.587s (2.587s including waiting). Image size: 154660917 bytes. 64m Normal Created pod/odh-observability-584f99c788-j2dld Container created 64m Normal Started pod/odh-observability-584f99c788-j2dld Container started 63m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-5cfb77bf8c Created pod: kserve-module-controller-manager-5cfb77bf8c-rgk8v 63m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-5cfb77bf8c from 0 to 1 63m Normal AddedInterface pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Add eth0 [10.132.0.34/23] from ovn-kubernetes 63m Normal Pulling pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 63m Normal Created pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Container created 63m Normal Pulled pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.449s (1.449s including waiting). Image size: 195729000 bytes. 63m Normal Started pod/kserve-module-controller-manager-5cfb77bf8c-rgk8v Container started 63m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-5cfb77bf8c-rgk8v_6e5dc507-f051-4f59-8ccb-e621809ac671 became leader 63m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-nt54h 63m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-jt9wq 63m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 63m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-f87fcf56c from 0 to 1 63m Normal SuccessfulCreate replicaset/odh-model-controller-f87fcf56c Created pod: odh-model-controller-f87fcf56c-zhx25 63m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-6strf 63m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 63m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 63m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-6strf Add eth0 [10.133.0.26/23] from ovn-kubernetes 63m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 63m Normal Pulling pod/odh-model-controller-f87fcf56c-zhx25 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 63m Normal AddedInterface pod/model-serving-api-75bd89fdfc-nt54h Add eth0 [10.133.0.28/23] from ovn-kubernetes 63m Normal Pulling pod/model-serving-api-75bd89fdfc-nt54h Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 63m Normal AddedInterface pod/odh-model-controller-f87fcf56c-zhx25 Add eth0 [10.133.0.29/23] from ovn-kubernetes 63m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Add eth0 [10.133.0.27/23] from ovn-kubernetes 63m Normal Pulling pod/kserve-controller-manager-776cc66d47-6strf Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 63m Normal Started pod/model-serving-api-75bd89fdfc-nt54h Container started 63m Normal Pulled pod/model-serving-api-75bd89fdfc-nt54h Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 3.073s (3.073s including waiting). Image size: 184925761 bytes. 63m Normal Created pod/model-serving-api-75bd89fdfc-nt54h Container created 63m Normal Pulled pod/kserve-controller-manager-776cc66d47-6strf Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 4.155s (4.155s including waiting). Image size: 247676581 bytes. 63m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Container started 63m Normal Pulled pod/odh-model-controller-f87fcf56c-zhx25 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 4.166s (4.166s including waiting). Image size: 236755567 bytes. 63m Normal Created pod/odh-model-controller-f87fcf56c-zhx25 Container created 63m Normal Started pod/odh-model-controller-f87fcf56c-zhx25 Container started 63m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 4.067s (4.067s including waiting). Image size: 235100981 bytes. 63m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-6strf_2a6e8497-e515-44f4-9d78-9a3e73b33310 became leader 63m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-f87fcf56c-zhx25_e96c1b1c-a24f-4fd1-a8d1-e1ed864cf963 became leader 63m Normal Created pod/kserve-controller-manager-776cc66d47-6strf Container created 63m Normal Started pod/kserve-controller-manager-776cc66d47-6strf Container started 63m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Container created 63m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-jt9wq_ed3f9694-ed17-4962-9bc6-3287757f04ef became leader 60m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-577f596567 from 0 to 1 60m Normal SuccessfulCreate replicaset/maas-controller-577f596567 Created pod: maas-controller-577f596567-qg4fp 60m Normal AddedInterface pod/maas-controller-78487d6d64-hdjk9 Add eth0 [10.132.0.51/23] from ovn-kubernetes 60m Normal Pulling pod/maas-controller-78487d6d64-hdjk9 Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 60m Normal SuccessfulCreate replicaset/maas-controller-78487d6d64 Created pod: maas-controller-78487d6d64-hdjk9 60m Normal AddedInterface pod/maas-controller-577f596567-qg4fp Add eth0 [10.132.0.50/23] from ovn-kubernetes 60m Normal Pulling pod/maas-controller-577f596567-qg4fp Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 60m Normal Created pod/maas-controller-577f596567-qg4fp Container created 60m Normal Pulled pod/maas-controller-577f596567-qg4fp Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.777s (2.777s including waiting). Image size: 199691795 bytes. 60m Normal Started pod/maas-controller-577f596567-qg4fp Container started 60m Normal Pulled pod/maas-controller-78487d6d64-hdjk9 Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.619s (2.619s including waiting). Image size: 199691795 bytes. 60m Normal Created pod/maas-controller-78487d6d64-hdjk9 Container created 60m Normal Started pod/maas-controller-78487d6d64-hdjk9 Container started 60m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-577f596567-qg4fp_71b0a7dd-9dbd-4152-8ad3-af048cadb28f became leader 60m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-7d94f55f4b from 0 to 1 60m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-7d94f55f4b Created pod: usage-logs-tenancy-proxy-7d94f55f4b-b8bbs 60m Warning FailedMount pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 60m Normal AddedInterface pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Add eth0 [10.132.0.52/23] from ovn-kubernetes 60m Normal Pulling pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 60m Normal SuccessfulDelete replicaset/maas-controller-577f596567 Deleted pod: maas-controller-577f596567-qg4fp 60m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-577f596567 from 1 to 0 60m Normal Killing pod/maas-controller-577f596567-qg4fp Stopping container manager 59m Normal Created pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Container created 59m Normal Pulled pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.154s (10.154s including waiting). Image size: 1089502687 bytes. 59m Normal Started pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Container started 59m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78487d6d64-hdjk9_771fccc0-e2a6-424c-888c-3c386bfeec94 became leader 59m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-54btj Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 59m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-54btj Add eth0 [10.133.0.32/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-54btj 59m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 59m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-54btj Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 343ms (343ms including waiting). Image size: 235100981 bytes. 59m Normal Created pod/llmisvc-controller-manager-6466c554d8-54btj Container created 59m Normal Started pod/llmisvc-controller-manager-6466c554d8-54btj Container started 58m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-jt9wq Stopping container manager 58m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-jt9wq 58m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 58m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-54btj_253117a4-9b3e-4d0a-ac32-34c526afbbbc became leader 58m Normal SuccessfulCreate replicaset/ai-gateway-controller-66dd76bf94 Created pod: ai-gateway-controller-66dd76bf94-lsdd7 58m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-66dd76bf94 from 0 to 1 58m Normal Pulling pod/ai-gateway-controller-66dd76bf94-lsdd7 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-202" 58m Normal AddedInterface pod/ai-gateway-controller-66dd76bf94-lsdd7 Add eth0 [10.132.0.58/23] from ovn-kubernetes 58m Normal Created pod/ai-gateway-controller-66dd76bf94-lsdd7 Container created 58m Normal Pulled pod/ai-gateway-controller-66dd76bf94-lsdd7 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-202" in 2.674s (2.674s including waiting). Image size: 183908754 bytes. 58m Normal Started pod/ai-gateway-controller-66dd76bf94-lsdd7 Container started 58m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-66dd76bf94-lsdd7_f8692769-cb4f-44a6-9058-7291cec80974 became leader 57m Normal Pulling pod/ai-gateway-controller-575996c6dd-2tngr Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-202" 57m Normal AddedInterface pod/ai-gateway-controller-575996c6dd-2tngr Add eth0 [10.132.0.61/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/ai-gateway-controller-575996c6dd Created pod: ai-gateway-controller-575996c6dd-2tngr 57m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-575996c6dd from 0 to 1 57m Normal Started pod/ai-gateway-controller-575996c6dd-2tngr Container started 57m Normal Created pod/ai-gateway-controller-575996c6dd-2tngr Container created 57m Normal Pulled pod/ai-gateway-controller-575996c6dd-2tngr Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-202" in 639ms (639ms including waiting). Image size: 183908754 bytes. 57m Normal SuccessfulDelete replicaset/ai-gateway-controller-66dd76bf94 Deleted pod: ai-gateway-controller-66dd76bf94-lsdd7 57m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-66dd76bf94 from 1 to 0 57m Normal Killing pod/ai-gateway-controller-66dd76bf94-lsdd7 Stopping container manager 57m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-575996c6dd-2tngr_b53eac4d-0111-496d-a259-acb43c2c9380 became leader 51m Warning Unhealthy pod/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs Readiness probe failed: Get "https://10.132.0.52:8443/ready": context deadline exceeded 51m Warning Unhealthy pod/maas-controller-78487d6d64-hdjk9 Liveness probe failed: Get "http://10.132.0.51:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal Pulling pod/test-curl-8644-cc17f3 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 50m Normal AddedInterface pod/test-curl-8644-cc17f3 Add eth0 [10.134.0.110/23] from ovn-kubernetes 50m Normal Pulled pod/test-curl-8644-cc17f3 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.217s (2.217s including waiting). Image size: 106100986 bytes. 50m Normal Created pod/test-curl-8644-cc17f3 Container created 50m Normal Started pod/test-curl-8644-cc17f3 Container started 50m Normal Started pod/test-curl-8644-dd7736 Container started 50m Normal AddedInterface pod/test-curl-8644-dd7736 Add eth0 [10.134.0.112/23] from ovn-kubernetes 50m Normal Pulled pod/test-curl-8644-dd7736 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 50m Normal Created pod/test-curl-8644-dd7736 Container created 50m Normal Killing pod/test-curl-8644-cc17f3 Stopping container test-curl-8644-cc17f3 50m Normal Killing pod/test-curl-8644-dd7736 Stopping container test-curl-8644-dd7736 46m Normal AddedInterface pod/test-curl-8659-a6d87b Add eth0 [10.132.0.129/23] from ovn-kubernetes 46m Normal Pulling pod/test-curl-8659-a6d87b Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 46m Normal Pulled pod/test-curl-8659-a6d87b Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 8.554s (8.554s including waiting). Image size: 106100986 bytes. 46m Normal Created pod/test-curl-8659-a6d87b Container created 46m Normal Started pod/test-curl-8659-a6d87b Container started 46m Normal Killing pod/test-curl-8659-a6d87b Stopping container test-curl-8659-a6d87b 43m Normal Pulled pod/test-curl-8659-bb37db Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/test-curl-8659-bb37db Add eth0 [10.132.0.146/23] from ovn-kubernetes 43m Normal Created pod/test-curl-8659-bb37db Container created 43m Normal Started pod/test-curl-8659-bb37db Container started 43m Normal Killing pod/test-curl-8659-bb37db Stopping container test-curl-8659-bb37db 42m Normal AddedInterface pod/test-curl-8653-0e6800 Add eth0 [10.132.0.148/23] from ovn-kubernetes 42m Normal Started pod/test-curl-8653-0e6800 Container started 42m Normal Created pod/test-curl-8653-0e6800 Container created 42m Normal Pulled pod/test-curl-8653-0e6800 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/test-curl-8653-0aba40 Add eth0 [10.132.0.149/23] from ovn-kubernetes 42m Normal Pulled pod/test-curl-8653-0aba40 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal Created pod/test-curl-8653-0aba40 Container created 42m Normal Started pod/test-curl-8653-0aba40 Container started 42m Normal Killing pod/test-curl-8653-0e6800 Stopping container test-curl-8653-0e6800 42m Normal AddedInterface pod/test-curl-8653-868444 Add eth0 [10.132.0.150/23] from ovn-kubernetes 42m Normal Killing pod/test-curl-8653-0aba40 Stopping container test-curl-8653-0aba40 42m Normal Started pod/test-curl-8653-868444 Container started 42m Normal Created pod/test-curl-8653-868444 Container created 42m Normal Pulled pod/test-curl-8653-868444 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal Killing pod/test-curl-8653-868444 Stopping container test-curl-8653-868444 42m Normal Started pod/test-curl-8653-c4fb43 Container started 42m Normal AddedInterface pod/test-curl-8653-c4fb43 Add eth0 [10.132.0.151/23] from ovn-kubernetes 42m Normal Pulled pod/test-curl-8653-c4fb43 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal Created pod/test-curl-8653-c4fb43 Container created 42m Normal Killing pod/test-curl-8653-c4fb43 Stopping container test-curl-8653-c4fb43 31m Normal AddedInterface pod/test-curl-8656-755369 Add eth0 [10.134.0.254/23] from ovn-kubernetes 31m Normal Pulled pod/test-curl-8656-755369 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 31m Normal Created pod/test-curl-8656-755369 Container created 31m Normal Started pod/test-curl-8656-755369 Container started 31m Normal Killing pod/test-curl-8656-755369 Stopping container test-curl-8656-755369 30m Normal Started pod/test-curl-8650-0edcad Container started 30m Normal AddedInterface pod/test-curl-8650-0edcad Add eth0 [10.132.0.195/23] from ovn-kubernetes 30m Normal Pulled pod/test-curl-8650-0edcad Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal Created pod/test-curl-8650-0edcad Container created 30m Normal Pulled pod/test-curl-8650-476664 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal Killing pod/test-curl-8650-0edcad Stopping container test-curl-8650-0edcad 30m Normal AddedInterface pod/test-curl-8650-476664 Add eth0 [10.132.0.196/23] from ovn-kubernetes 30m Normal Created pod/test-curl-8650-476664 Container created 30m Normal Started pod/test-curl-8650-476664 Container started 29m Normal Killing pod/test-curl-8650-476664 Stopping container test-curl-8650-476664 14m Normal SuccessfulDelete replicaset/maas-controller-78487d6d64 Deleted pod: maas-controller-78487d6d64-hdjk9 14m Normal Killing pod/maas-controller-78487d6d64-hdjk9 Stopping container manager 14m Normal Pulling pod/maas-controller-78487d6d64-fkx6f Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 14m Normal AddedInterface pod/maas-controller-78487d6d64-fkx6f Add eth0 [10.132.0.241/23] from ovn-kubernetes 14m Normal SuccessfulCreate replicaset/maas-controller-78487d6d64 Created pod: maas-controller-78487d6d64-fkx6f 14m Normal Pulled pod/maas-controller-78487d6d64-fkx6f Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 510ms (510ms including waiting). Image size: 199691795 bytes. 14m Normal Started pod/maas-controller-78487d6d64-fkx6f Container started 14m Normal Created pod/maas-controller-78487d6d64-fkx6f Container created 14m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78487d6d64-fkx6f_05fe601b-25b8-47d0-91ad-882fb3e84c3d became leader 13m Normal Killing pod/maas-controller-78487d6d64-fkx6f Stopping container manager 13m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-78487d6d64 from 1 to 0 13m Normal SuccessfulDelete replicaset/maas-controller-78487d6d64 Deleted pod: maas-controller-78487d6d64-fkx6f 13m Normal Pulling pod/maas-controller-78487d6d64-k525h Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal SuccessfulCreate replicaset/maas-controller-78487d6d64 Created pod: maas-controller-78487d6d64-k525h 13m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-78487d6d64 from 0 to 1 13m Normal AddedInterface pod/maas-controller-78487d6d64-k525h Add eth0 [10.132.0.243/23] from ovn-kubernetes 13m Normal Started pod/maas-controller-78487d6d64-k525h Container started 13m Normal Created pod/maas-controller-78487d6d64-k525h Container created 13m Normal Pulled pod/maas-controller-78487d6d64-k525h Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 322ms (322ms including waiting). Image size: 199691795 bytes. 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-78487d6d64-k525h_5dce7c0b-c977-4804-8e80-2a06b3b521d9 became leader