=== events in opendatahub (2026-10-29T23:21:26Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 13m Normal Scheduled pod/maas-controller-86789cb564-4vgtn Successfully assigned opendatahub/maas-controller-86789cb564-4vgtn to ip-10-0-133-251.ec2.internal 54m Normal Scheduled pod/ai-gateway-controller-55b4c89cbd-2tckw Successfully assigned opendatahub/ai-gateway-controller-55b4c89cbd-2tckw to ip-10-0-133-251.ec2.internal 56m Normal Scheduled pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Successfully assigned opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 to ip-10-0-133-251.ec2.internal 47m Normal Scheduled pod/test-curl-9023-d8c18c Successfully assigned opendatahub/test-curl-9023-d8c18c to ip-10-0-134-94.ec2.internal 26m Normal Scheduled pod/test-curl-9023-ccd781 Successfully assigned opendatahub/test-curl-9023-ccd781 to ip-10-0-134-94.ec2.internal 26m Normal Scheduled pod/test-curl-9023-60de05 Successfully assigned opendatahub/test-curl-9023-60de05 to ip-10-0-134-94.ec2.internal 47m Normal Scheduled pod/test-curl-9023-101e58 Successfully assigned opendatahub/test-curl-9023-101e58 to ip-10-0-134-94.ec2.internal 41m Normal Scheduled pod/test-curl-9020-b267e7 Successfully assigned opendatahub/test-curl-9020-b267e7 to ip-10-0-133-251.ec2.internal 37m Normal Scheduled pod/test-curl-9020-3e647b Successfully assigned opendatahub/test-curl-9020-3e647b to ip-10-0-133-251.ec2.internal 30m Normal Scheduled pod/test-curl-9011-dedc99 Successfully assigned opendatahub/test-curl-9011-dedc99 to ip-10-0-134-94.ec2.internal 54m Normal Scheduled pod/ai-gateway-controller-85bb745f44-xt49q Successfully assigned opendatahub/ai-gateway-controller-85bb745f44-xt49q to ip-10-0-139-29.ec2.internal 41m Normal Scheduled pod/test-curl-9005-bbf3c9 Successfully assigned opendatahub/test-curl-9005-bbf3c9 to ip-10-0-133-251.ec2.internal 41m Normal Scheduled pod/test-curl-9005-a8a6ca Successfully assigned opendatahub/test-curl-9005-a8a6ca to ip-10-0-133-251.ec2.internal 41m Normal Scheduled pod/test-curl-9005-2ede4e Successfully assigned opendatahub/test-curl-9005-2ede4e to ip-10-0-133-251.ec2.internal 41m Normal Scheduled pod/test-curl-9005-2ae7b8 Successfully assigned opendatahub/test-curl-9005-2ae7b8 to ip-10-0-133-251.ec2.internal 61m Normal Scheduled pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Successfully assigned opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m to ip-10-0-133-251.ec2.internal 61m Normal Scheduled pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Successfully assigned opendatahub/opendatahub-operator-controller-manager-b95c7d58c-nrbsf to ip-10-0-134-94.ec2.internal 56m Normal Scheduled pod/maas-controller-577f596567-brs7d Successfully assigned opendatahub/maas-controller-577f596567-brs7d to ip-10-0-133-251.ec2.internal 59m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-mgc6m to ip-10-0-134-94.ec2.internal 55m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-ldwdt Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-ldwdt to ip-10-0-134-94.ec2.internal 56m Normal Scheduled pod/maas-controller-86789cb564-phf64 Successfully assigned opendatahub/maas-controller-86789cb564-phf64 to ip-10-0-133-251.ec2.internal 60m Normal Scheduled pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Successfully assigned opendatahub/kserve-module-controller-manager-5cfb77bf8c-lw9wk to ip-10-0-134-94.ec2.internal 59m Normal Scheduled pod/kserve-controller-manager-776cc66d47-9zlw9 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-9zlw9 to ip-10-0-134-94.ec2.internal 60m Normal Scheduled pod/odh-observability-584f99c788-nk5rv Successfully assigned opendatahub/odh-observability-584f99c788-nk5rv to ip-10-0-133-251.ec2.internal 59m Normal Scheduled pod/odh-model-controller-f87fcf56c-2fxfz Successfully assigned opendatahub/odh-model-controller-f87fcf56c-2fxfz to ip-10-0-134-94.ec2.internal 59m Normal Scheduled pod/model-serving-api-75bd89fdfc-dg9sk Successfully assigned opendatahub/model-serving-api-75bd89fdfc-dg9sk to ip-10-0-134-94.ec2.internal 61m Normal Scheduled pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Successfully assigned opendatahub/opendatahub-operator-controller-manager-b95c7d58c-cv5kd to ip-10-0-139-29.ec2.internal 13m Normal Scheduled pod/maas-controller-86789cb564-nmrx6 Successfully assigned opendatahub/maas-controller-86789cb564-nmrx6 to ip-10-0-133-251.ec2.internal 61m Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 requirements not yet checked 61m Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 one or more requirements couldn't be found 61m Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 all requirements found, attempting install 61m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-b95c7d58c Created pod: opendatahub-operator-controller-manager-b95c7d58c-cv5kd 61m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-b95c7d58c Created pod: opendatahub-operator-controller-manager-b95c7d58c-vzk8m 61m Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-b95c7d58c from 0 to 3 61m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-b95c7d58c Created pod: opendatahub-operator-controller-manager-b95c7d58c-nrbsf 61m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 waiting for install components to report healthy 61m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal AddedInterface pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Add eth0 [10.134.0.15/23] from ovn-kubernetes 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal AddedInterface pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Add eth0 [10.132.0.34/23] from ovn-kubernetes 61m Normal AddedInterface pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Add eth0 [10.133.0.25/23] from ovn-kubernetes 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.652s (2.652s including waiting). Image size: 273709574 bytes. 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.703s (2.703s including waiting). Image size: 273709574 bytes. 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.706s (2.706s including waiting). Image size: 273709574 bytes. 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Container created 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Container started 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Container started 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Container created 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Container started 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Container created 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 251ms (251ms including waiting). Image size: 273709574 bytes. 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Container created 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Container started 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 313ms (313ms including waiting). Image size: 273709574 bytes. 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-nrbsf Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal Pulled pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 299ms (299ms including waiting). Image size: 273709574 bytes. 61m Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-b95c7d58c-nrbsf_319d7a60-e2b1-4092-9114-b412da121287 became leader 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Container created 61m Normal Pulling pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Container started 61m Normal Created pod/opendatahub-operator-controller-manager-b95c7d58c-cv5kd Container created 61m Normal Started pod/opendatahub-operator-controller-manager-b95c7d58c-vzk8m Container started 60m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 install strategy completed with no errors 60m Normal Generated certificate/odh-observability-webhook-cert Stored new private key in temporary Secret resource "odh-observability-webhook-cert-4wkhq" 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal SuccessfulCreate replicaset/odh-observability-584f99c788 Created pod: odh-observability-584f99c788-nk5rv 60m Normal Issuing certificate/odh-observability-webhook-cert The certificate has been successfully issued 60m Normal Requested certificate/odh-observability-webhook-cert Created new CertificateRequest resource "odh-observability-webhook-cert-1" 60m Normal CertificateIssued certificaterequest/odh-observability-webhook-cert-1 Certificate fetched from issuer successfully 60m Normal Issuing certificate/odh-observability-webhook-cert Issuing certificate as Secret does not exist 60m Normal ScalingReplicaSet deployment/odh-observability Scaled up replica set odh-observability-584f99c788 from 0 to 1 60m Warning FailedMount pod/odh-observability-584f99c788-nk5rv MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 60m Normal cert-manager.io certificaterequest/odh-observability-webhook-cert-1 Certificate request has been approved by cert-manager.io 60m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 60m Normal AddedInterface pod/odh-observability-584f99c788-nk5rv Add eth0 [10.134.0.18/23] from ovn-kubernetes 60m Normal Pulling pod/odh-observability-584f99c788-nk5rv Pulling image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" 60m Normal Pulled pod/odh-observability-584f99c788-nk5rv Successfully pulled image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" in 2.306s (2.306s including waiting). Image size: 154660917 bytes. 60m Normal Created pod/odh-observability-584f99c788-nk5rv Container created 60m Normal Started pod/odh-observability-584f99c788-nk5rv Container started 60m Normal AddedInterface pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Add eth0 [10.132.0.37/23] from ovn-kubernetes 60m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-5cfb77bf8c from 0 to 1 60m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-5cfb77bf8c Created pod: kserve-module-controller-manager-5cfb77bf8c-lw9wk 60m Normal Pulling pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 60m Normal Pulled pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.246s (2.246s including waiting). Image size: 195729000 bytes. 60m Normal Started pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Container started 60m Normal Created pod/kserve-module-controller-manager-5cfb77bf8c-lw9wk Container created 60m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-5cfb77bf8c-lw9wk_7edb7370-b0c8-41d8-b8f4-57715ef95124 became leader 59m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-9zlw9 59m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 59m Normal SuccessfulCreate replicaset/odh-model-controller-f87fcf56c Created pod: odh-model-controller-f87fcf56c-2fxfz 59m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Add eth0 [10.132.0.39/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-mgc6m 59m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-f87fcf56c from 0 to 1 59m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-9zlw9 Add eth0 [10.132.0.38/23] from ovn-kubernetes 59m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 59m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-dg9sk 59m Normal Pulling pod/odh-model-controller-f87fcf56c-2fxfz Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 59m Normal AddedInterface pod/odh-model-controller-f87fcf56c-2fxfz Add eth0 [10.132.0.41/23] from ovn-kubernetes 59m Normal AddedInterface pod/model-serving-api-75bd89fdfc-dg9sk Add eth0 [10.132.0.40/23] from ovn-kubernetes 59m Normal Pulling pod/model-serving-api-75bd89fdfc-dg9sk Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 59m Normal Pulling pod/kserve-controller-manager-776cc66d47-9zlw9 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 59m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 59m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 59m Normal Started pod/model-serving-api-75bd89fdfc-dg9sk Container started 59m Normal Created pod/kserve-controller-manager-776cc66d47-9zlw9 Container created 59m Normal Pulled pod/model-serving-api-75bd89fdfc-dg9sk Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.492s (2.492s including waiting). Image size: 184925761 bytes. 59m Normal Pulled pod/kserve-controller-manager-776cc66d47-9zlw9 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.139s (3.139s including waiting). Image size: 247676581 bytes. 59m Normal Started pod/kserve-controller-manager-776cc66d47-9zlw9 Container started 59m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Container started 59m Normal Created pod/model-serving-api-75bd89fdfc-dg9sk Container created 59m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Container created 59m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.669s (2.669s including waiting). Image size: 235100981 bytes. 59m Normal Created pod/odh-model-controller-f87fcf56c-2fxfz Container created 59m Normal Started pod/odh-model-controller-f87fcf56c-2fxfz Container started 59m Normal Pulled pod/odh-model-controller-f87fcf56c-2fxfz Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.516s (3.516s including waiting). Image size: 236755567 bytes. 59m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-9zlw9_46d3e1ef-9dd3-4eff-ba7e-a38777bebba5 became leader 59m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-f87fcf56c-2fxfz_514aa8e6-b5b2-4025-9a2d-e60f6882fea4 became leader 59m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-mgc6m_be05fd90-33aa-4350-a94b-327b4c8d5aa6 became leader 56m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-577f596567 from 0 to 1 56m Normal SuccessfulCreate replicaset/maas-controller-577f596567 Created pod: maas-controller-577f596567-brs7d 56m Normal Pulling pod/maas-controller-86789cb564-phf64 Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 56m Normal AddedInterface pod/maas-controller-86789cb564-phf64 Add eth0 [10.134.0.32/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/maas-controller-86789cb564 Created pod: maas-controller-86789cb564-phf64 56m Warning FailedMount pod/maas-controller-577f596567-brs7d MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 56m Normal Pulling pod/maas-controller-577f596567-brs7d Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 56m Normal AddedInterface pod/maas-controller-577f596567-brs7d Add eth0 [10.134.0.31/23] from ovn-kubernetes 56m Normal Started pod/maas-controller-86789cb564-phf64 Container started 56m Normal Pulled pod/maas-controller-577f596567-brs7d Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.859s (2.859s including waiting). Image size: 199691795 bytes. 56m Normal Pulled pod/maas-controller-86789cb564-phf64 Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.798s (2.798s including waiting). Image size: 199691795 bytes. 56m Normal Created pod/maas-controller-86789cb564-phf64 Container created 56m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-577f596567-brs7d_6edb83d6-8334-4009-8b4a-4b8e4eac5d09 became leader 56m Normal Created pod/maas-controller-577f596567-brs7d Container created 56m Normal Started pod/maas-controller-577f596567-brs7d Container started 56m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-7d94f55f4b from 0 to 1 56m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-7d94f55f4b Created pod: usage-logs-tenancy-proxy-7d94f55f4b-rdht5 56m Normal Pulling pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 56m Normal AddedInterface pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Add eth0 [10.134.0.33/23] from ovn-kubernetes 56m Normal Killing pod/maas-controller-577f596567-brs7d Stopping container manager 56m Normal SuccessfulDelete replicaset/maas-controller-577f596567 Deleted pod: maas-controller-577f596567-brs7d 56m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-577f596567 from 1 to 0 56m Normal Pulled pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.095s (10.095s including waiting). Image size: 1089502687 bytes. 56m Normal Started pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Container started 56m Normal Created pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Container created 56m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-86789cb564-phf64_77feaa8a-360e-4e8c-a687-a9acd50e8d52 became leader 55m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-ldwdt Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 55m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-ldwdt 55m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 55m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-ldwdt Add eth0 [10.132.0.44/23] from ovn-kubernetes 55m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-ldwdt Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 365ms (365ms including waiting). Image size: 235100981 bytes. 55m Normal Started pod/llmisvc-controller-manager-6466c554d8-ldwdt Container started 55m Normal Created pod/llmisvc-controller-manager-6466c554d8-ldwdt Container created 55m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-mgc6m Stopping container manager 55m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-mgc6m 55m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 55m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-ldwdt_5bc97a29-2f93-4068-aa50-1a613ce7c0c1 became leader 54m Normal SuccessfulCreate replicaset/ai-gateway-controller-55b4c89cbd Created pod: ai-gateway-controller-55b4c89cbd-2tckw 54m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-55b4c89cbd from 0 to 1 54m Normal AddedInterface pod/ai-gateway-controller-55b4c89cbd-2tckw Add eth0 [10.134.0.38/23] from ovn-kubernetes 54m Normal Pulling pod/ai-gateway-controller-55b4c89cbd-2tckw Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" 54m Normal Started pod/ai-gateway-controller-55b4c89cbd-2tckw Container started 54m Normal Created pod/ai-gateway-controller-55b4c89cbd-2tckw Container created 54m Normal Pulled pod/ai-gateway-controller-55b4c89cbd-2tckw Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" in 2.327s (2.327s including waiting). Image size: 183926180 bytes. 54m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-55b4c89cbd-2tckw_b5f192ac-6f52-4bf3-b5f1-3d2d79bf4ef5 became leader 54m Normal SuccessfulCreate replicaset/ai-gateway-controller-85bb745f44 Created pod: ai-gateway-controller-85bb745f44-xt49q 54m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-85bb745f44 from 0 to 1 54m Normal Pulling pod/ai-gateway-controller-85bb745f44-xt49q Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" 54m Normal AddedInterface pod/ai-gateway-controller-85bb745f44-xt49q Add eth0 [10.133.0.45/23] from ovn-kubernetes 54m Normal Created pod/ai-gateway-controller-85bb745f44-xt49q Container created 54m Normal Started pod/ai-gateway-controller-85bb745f44-xt49q Container started 54m Normal Pulled pod/ai-gateway-controller-85bb745f44-xt49q Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-190" in 2.38s (2.38s including waiting). Image size: 183926180 bytes. 54m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-55b4c89cbd from 1 to 0 54m Normal SuccessfulDelete replicaset/ai-gateway-controller-55b4c89cbd Deleted pod: ai-gateway-controller-55b4c89cbd-2tckw 54m Normal Killing pod/ai-gateway-controller-55b4c89cbd-2tckw Stopping container manager 53m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-85bb745f44-xt49q_3e83d4f9-3e9e-4ff7-b4e7-2e26e35549ae became leader 47m Normal Pulling pod/test-curl-9023-d8c18c Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 47m Normal AddedInterface pod/test-curl-9023-d8c18c Add eth0 [10.132.0.93/23] from ovn-kubernetes 47m Normal Started pod/test-curl-9023-d8c18c Container started 47m Normal Created pod/test-curl-9023-d8c18c Container created 47m Normal Pulled pod/test-curl-9023-d8c18c Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 484ms (484ms including waiting). Image size: 106100986 bytes. 47m Normal AddedInterface pod/test-curl-9023-101e58 Add eth0 [10.132.0.94/23] from ovn-kubernetes 47m Normal Started pod/test-curl-9023-101e58 Container started 47m Normal Created pod/test-curl-9023-101e58 Container created 47m Normal Pulled pod/test-curl-9023-101e58 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 47m Normal Killing pod/test-curl-9023-d8c18c Stopping container test-curl-9023-d8c18c 47m Normal Killing pod/test-curl-9023-101e58 Stopping container test-curl-9023-101e58 41m Normal Pulling pod/test-curl-9020-b267e7 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 41m Normal AddedInterface pod/test-curl-9020-b267e7 Add eth0 [10.134.0.114/23] from ovn-kubernetes 41m Normal Pulled pod/test-curl-9020-b267e7 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 3.381s (3.381s including waiting). Image size: 106100986 bytes. 41m Normal Created pod/test-curl-9020-b267e7 Container created 41m Normal Started pod/test-curl-9020-b267e7 Container started 41m Normal Killing pod/test-curl-9020-b267e7 Stopping container test-curl-9020-b267e7 41m Normal AddedInterface pod/test-curl-9005-bbf3c9 Add eth0 [10.134.0.125/23] from ovn-kubernetes 41m Normal Pulled pod/test-curl-9005-bbf3c9 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal Started pod/test-curl-9005-bbf3c9 Container started 41m Normal Created pod/test-curl-9005-bbf3c9 Container created 41m Normal Killing pod/test-curl-9005-bbf3c9 Stopping container test-curl-9005-bbf3c9 41m Normal AddedInterface pod/test-curl-9005-2ae7b8 Add eth0 [10.134.0.130/23] from ovn-kubernetes 41m Normal Pulled pod/test-curl-9005-2ae7b8 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal Started pod/test-curl-9005-2ae7b8 Container started 41m Normal Created pod/test-curl-9005-2ae7b8 Container created 41m Normal AddedInterface pod/test-curl-9005-a8a6ca Add eth0 [10.134.0.132/23] from ovn-kubernetes 41m Normal Killing pod/test-curl-9005-2ae7b8 Stopping container test-curl-9005-2ae7b8 41m Normal Pulled pod/test-curl-9005-a8a6ca Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal Created pod/test-curl-9005-a8a6ca Container created 41m Normal Started pod/test-curl-9005-a8a6ca Container started 41m Normal AddedInterface pod/test-curl-9005-2ede4e Add eth0 [10.134.0.133/23] from ovn-kubernetes 41m Normal Started pod/test-curl-9005-2ede4e Container started 41m Normal Created pod/test-curl-9005-2ede4e Container created 41m Normal Pulled pod/test-curl-9005-2ede4e Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal Killing pod/test-curl-9005-a8a6ca Stopping container test-curl-9005-a8a6ca 41m Normal Killing pod/test-curl-9005-2ede4e Stopping container test-curl-9005-2ede4e 38m Warning Unhealthy pod/usage-logs-tenancy-proxy-7d94f55f4b-rdht5 Readiness probe failed: Get "https://10.134.0.33:8443/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 37m Normal AddedInterface pod/test-curl-9020-3e647b Add eth0 [10.134.0.152/23] from ovn-kubernetes 37m Normal Pulled pod/test-curl-9020-3e647b Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 37m Normal Created pod/test-curl-9020-3e647b Container created 37m Normal Started pod/test-curl-9020-3e647b Container started 37m Normal Killing pod/test-curl-9020-3e647b Stopping container test-curl-9020-3e647b 30m Normal Started pod/test-curl-9011-dedc99 Container started 30m Normal Created pod/test-curl-9011-dedc99 Container created 30m Normal Pulled pod/test-curl-9011-dedc99 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/test-curl-9011-dedc99 Add eth0 [10.132.0.148/23] from ovn-kubernetes 30m Normal Killing pod/test-curl-9011-dedc99 Stopping container test-curl-9011-dedc99 26m Normal AddedInterface pod/test-curl-9023-ccd781 Add eth0 [10.132.0.156/23] from ovn-kubernetes 26m Normal Started pod/test-curl-9023-ccd781 Container started 26m Normal Created pod/test-curl-9023-ccd781 Container created 26m Normal Pulled pod/test-curl-9023-ccd781 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/test-curl-9023-60de05 Add eth0 [10.132.0.157/23] from ovn-kubernetes 26m Normal Created pod/test-curl-9023-60de05 Container created 26m Normal Started pod/test-curl-9023-60de05 Container started 26m Normal Pulled pod/test-curl-9023-60de05 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 26m Normal Killing pod/test-curl-9023-ccd781 Stopping container test-curl-9023-ccd781 26m Normal Killing pod/test-curl-9023-60de05 Stopping container test-curl-9023-60de05 13m Normal SuccessfulDelete replicaset/maas-controller-86789cb564 Deleted pod: maas-controller-86789cb564-phf64 13m Normal Killing pod/maas-controller-86789cb564-phf64 Stopping container manager 13m Normal SuccessfulCreate replicaset/maas-controller-86789cb564 Created pod: maas-controller-86789cb564-4vgtn 13m Normal Pulling pod/maas-controller-86789cb564-4vgtn Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal AddedInterface pod/maas-controller-86789cb564-4vgtn Add eth0 [10.134.0.255/23] from ovn-kubernetes 13m Normal Created pod/maas-controller-86789cb564-4vgtn Container created 13m Normal Pulled pod/maas-controller-86789cb564-4vgtn Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 494ms (494ms including waiting). Image size: 199691795 bytes. 13m Normal Started pod/maas-controller-86789cb564-4vgtn Container started 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-86789cb564-4vgtn_9e2237ec-f3e9-46a7-b210-8df3a9977689 became leader 13m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-86789cb564 from 1 to 0 13m Normal Killing pod/maas-controller-86789cb564-4vgtn Stopping container manager 13m Normal SuccessfulDelete replicaset/maas-controller-86789cb564 Deleted pod: maas-controller-86789cb564-4vgtn 13m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-86789cb564 from 0 to 1 13m Normal SuccessfulCreate replicaset/maas-controller-86789cb564 Created pod: maas-controller-86789cb564-nmrx6 13m Normal Pulling pod/maas-controller-86789cb564-nmrx6 Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal AddedInterface pod/maas-controller-86789cb564-nmrx6 Add eth0 [10.134.1.3/23] from ovn-kubernetes 13m Normal Pulled pod/maas-controller-86789cb564-nmrx6 Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 480ms (480ms including waiting). Image size: 199691795 bytes. 13m Normal Started pod/maas-controller-86789cb564-nmrx6 Container started 13m Normal Created pod/maas-controller-86789cb564-nmrx6 Container created 12m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-86789cb564-nmrx6_8c1b4aaa-3688-404c-b62b-f4ca20196244 became leader