=== events in opendatahub (2026-10-30T05:54:49Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 61m Normal Scheduled pod/maas-controller-577f596567-5pmj8 Successfully assigned opendatahub/maas-controller-577f596567-5pmj8 to ip-10-0-139-228.ec2.internal 61m Normal Scheduled pod/maas-controller-5d77b88d59-8t5sz Successfully assigned opendatahub/maas-controller-5d77b88d59-8t5sz to ip-10-0-139-228.ec2.internal 61m Normal Scheduled pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Successfully assigned opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/test-curl-8876-e8f1d5 Successfully assigned opendatahub/test-curl-8876-e8f1d5 to ip-10-0-135-98.ec2.internal 23m Normal Scheduled pod/test-curl-8876-8ac5bc Successfully assigned opendatahub/test-curl-8876-8ac5bc to ip-10-0-139-228.ec2.internal 23m Normal Scheduled pod/test-curl-8876-3723ff Successfully assigned opendatahub/test-curl-8876-3723ff to ip-10-0-139-228.ec2.internal 44m Normal Scheduled pod/test-curl-8876-02f73b Successfully assigned opendatahub/test-curl-8876-02f73b to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/test-curl-8867-fbfc64 Successfully assigned opendatahub/test-curl-8867-fbfc64 to ip-10-0-139-228.ec2.internal 13m Normal Scheduled pod/maas-controller-5d77b88d59-t8d5z Successfully assigned opendatahub/maas-controller-5d77b88d59-t8d5z to ip-10-0-139-228.ec2.internal 51m Normal Scheduled pod/test-curl-8867-539628 Successfully assigned opendatahub/test-curl-8867-539628 to ip-10-0-135-98.ec2.internal 59m Normal Scheduled pod/ai-gateway-controller-77449b8677-8jpjr Successfully assigned opendatahub/ai-gateway-controller-77449b8677-8jpjr to ip-10-0-134-105.ec2.internal 46m Normal Scheduled pod/test-curl-8867-4aad04 Successfully assigned opendatahub/test-curl-8867-4aad04 to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/test-curl-8867-280ef2 Successfully assigned opendatahub/test-curl-8867-280ef2 to ip-10-0-139-228.ec2.internal 47m Normal Scheduled pod/test-curl-8867-182852 Successfully assigned opendatahub/test-curl-8867-182852 to ip-10-0-139-228.ec2.internal 59m Normal Scheduled pod/ai-gateway-controller-5d9cd99f45-lwzf6 Successfully assigned opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6 to ip-10-0-134-105.ec2.internal 21m Normal Scheduled pod/test-curl-8864-f40e45 Successfully assigned opendatahub/test-curl-8864-f40e45 to ip-10-0-139-228.ec2.internal 51m Normal Scheduled pod/test-curl-8867-cde874 Successfully assigned opendatahub/test-curl-8867-cde874 to ip-10-0-135-98.ec2.internal 65m Normal Scheduled pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Successfully assigned opendatahub/opendatahub-operator-controller-manager-8484dd48c9-xqtxn to ip-10-0-139-228.ec2.internal 65m Normal Scheduled pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Successfully assigned opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8 to ip-10-0-135-98.ec2.internal 65m Normal Scheduled pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Successfully assigned opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 to ip-10-0-134-105.ec2.internal 60m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-9jkvx Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx to ip-10-0-135-98.ec2.internal 64m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-57mpg Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg to ip-10-0-135-98.ec2.internal 64m Normal Scheduled pod/kserve-controller-manager-776cc66d47-4jgtx Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-4jgtx to ip-10-0-135-98.ec2.internal 13m Normal Scheduled pod/maas-controller-5d77b88d59-2jptv Successfully assigned opendatahub/maas-controller-5d77b88d59-2jptv to ip-10-0-139-228.ec2.internal 65m Normal Scheduled pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Successfully assigned opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv to ip-10-0-135-98.ec2.internal 65m Normal Scheduled pod/odh-observability-584f99c788-gm2nc Successfully assigned opendatahub/odh-observability-584f99c788-gm2nc to ip-10-0-139-228.ec2.internal 64m Normal Scheduled pod/odh-model-controller-f87fcf56c-dp85z Successfully assigned opendatahub/odh-model-controller-f87fcf56c-dp85z to ip-10-0-134-105.ec2.internal 64m Normal Scheduled pod/model-serving-api-75bd89fdfc-t2c9k Successfully assigned opendatahub/model-serving-api-75bd89fdfc-t2c9k to ip-10-0-135-98.ec2.internal 65m Normal ScalingReplicaSet deployment/opendatahub-operator-controller-manager Scaled up replica set opendatahub-operator-controller-manager-8484dd48c9 from 0 to 3 65m Normal RequirementsUnknown clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 requirements not yet checked 65m Normal RequirementsNotMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 one or more requirements couldn't be found 65m Normal AllRequirementsMet clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 all requirements found, attempting install 65m Normal AddedInterface pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Add eth0 [10.133.0.23/23] from ovn-kubernetes 65m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8484dd48c9 Created pod: opendatahub-operator-controller-manager-8484dd48c9-xqtxn 65m Normal AddedInterface pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Add eth0 [10.132.0.32/23] from ovn-kubernetes 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 waiting for install components to report healthy 65m Normal AddedInterface pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Add eth0 [10.134.0.18/23] from ovn-kubernetes 65m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8484dd48c9 Created pod: opendatahub-operator-controller-manager-8484dd48c9-sxchv 65m Normal SuccessfulCreate replicaset/opendatahub-operator-controller-manager-8484dd48c9 Created pod: opendatahub-operator-controller-manager-8484dd48c9-2v6v6 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal InstallWaiting clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability. 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Container created 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Container started 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.695s (2.695s including waiting). Image size: 273709574 bytes. 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Container started 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Container created 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Container started 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.757s (2.757s including waiting). Image size: 273709574 bytes. 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Container created 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.664s (2.664s including waiting). Image size: 273709574 bytes. 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 353ms (353ms including waiting). Image size: 273709574 bytes. 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Container created 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Container started 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 332ms (332ms including waiting). Image size: 273709574 bytes. 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Container started 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Container created 65m Normal Pulling pod/opendatahub-operator-controller-manager-8484dd48c9-xqtxn Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" 65m Normal Pulled pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 304ms (304ms including waiting). Image size: 273709574 bytes. 65m Normal Created pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Container created 65m Normal Started pod/opendatahub-operator-controller-manager-8484dd48c9-sxchv Container started 65m Normal LeaderElection lease/07ed84f7.opendatahub.io opendatahub-operator-controller-manager-8484dd48c9-2v6v6_b896e2a0-d445-41ac-b228-14f14b0bfb5f became leader 65m Normal InstallSucceeded clusterserviceversion/opendatahub-operator.v3.6.0-ea.2 install strategy completed with no errors 65m Normal ScalingReplicaSet deployment/odh-observability Scaled up replica set odh-observability-584f99c788 from 0 to 1 65m Normal SuccessfulCreate replicaset/odh-observability-584f99c788 Created pod: odh-observability-584f99c788-gm2nc 65m Warning FailedMount pod/odh-observability-584f99c788-gm2nc MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 65m Normal Issuing certificate/odh-observability-webhook-cert Issuing certificate as Secret does not exist 65m Normal Generated certificate/odh-observability-webhook-cert Stored new private key in temporary Secret resource "odh-observability-webhook-cert-tmq84" 65m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 65m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 65m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 65m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 65m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 65m Normal WaitingForApproval certificaterequest/odh-observability-webhook-cert-1 Not signing CertificateRequest until it is Approved 65m Normal Requested certificate/odh-observability-webhook-cert Created new CertificateRequest resource "odh-observability-webhook-cert-1" 65m Normal cert-manager.io certificaterequest/odh-observability-webhook-cert-1 Certificate request has been approved by cert-manager.io 65m Normal CertificateIssued certificaterequest/odh-observability-webhook-cert-1 Certificate fetched from issuer successfully 65m Normal Issuing certificate/odh-observability-webhook-cert The certificate has been successfully issued 65m Normal AddedInterface pod/odh-observability-584f99c788-gm2nc Add eth0 [10.132.0.33/23] from ovn-kubernetes 65m Normal Pulling pod/odh-observability-584f99c788-gm2nc Pulling image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" 65m Normal Pulled pod/odh-observability-584f99c788-gm2nc Successfully pulled image "quay.io/opendatahub/odh-observability@sha256:03689e71fb8879ea696366b83d2a4727eb76e696baf8c2be2b18121c0534db4e" in 2.888s (2.888s including waiting). Image size: 154660917 bytes. 65m Normal Created pod/odh-observability-584f99c788-gm2nc Container created 65m Normal Started pod/odh-observability-584f99c788-gm2nc Container started 65m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-5cfb77bf8c Created pod: kserve-module-controller-manager-5cfb77bf8c-c28c8 65m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-5cfb77bf8c from 0 to 1 65m Normal Pulling pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 65m Normal AddedInterface pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Add eth0 [10.133.0.26/23] from ovn-kubernetes 65m Normal Created pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Container created 65m Normal Pulled pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.481s (2.481s including waiting). Image size: 195729000 bytes. 65m Normal Started pod/kserve-module-controller-manager-5cfb77bf8c-c28c8 Container started 65m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-5cfb77bf8c-c28c8_438b080d-5db6-41ee-9b39-2d3e1a66df25 became leader 64m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 64m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-f87fcf56c from 0 to 1 64m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 64m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-57mpg 64m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 64m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-4jgtx 64m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-t2c9k 64m Normal SuccessfulCreate replicaset/odh-model-controller-f87fcf56c Created pod: odh-model-controller-f87fcf56c-dp85z 64m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-57mpg Add eth0 [10.133.0.28/23] from ovn-kubernetes 64m Normal AddedInterface pod/model-serving-api-75bd89fdfc-t2c9k Add eth0 [10.133.0.29/23] from ovn-kubernetes 64m Normal Pulling pod/model-serving-api-75bd89fdfc-t2c9k Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 64m Normal Pulling pod/kserve-controller-manager-776cc66d47-4jgtx Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 64m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-57mpg Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 64m Normal AddedInterface pod/odh-model-controller-f87fcf56c-dp85z Add eth0 [10.134.0.21/23] from ovn-kubernetes 64m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-4jgtx Add eth0 [10.133.0.27/23] from ovn-kubernetes 64m Normal Pulling pod/odh-model-controller-f87fcf56c-dp85z Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 64m Normal Pulled pod/odh-model-controller-f87fcf56c-dp85z Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 2.515s (2.515s including waiting). Image size: 236755567 bytes. 64m Normal Started pod/kserve-controller-manager-776cc66d47-4jgtx Container started 64m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-57mpg Container started 64m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-57mpg Container created 64m Normal Created pod/odh-model-controller-f87fcf56c-dp85z Container created 64m Normal Started pod/odh-model-controller-f87fcf56c-dp85z Container started 64m Normal Started pod/model-serving-api-75bd89fdfc-t2c9k Container started 64m Normal Created pod/model-serving-api-75bd89fdfc-t2c9k Container created 64m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-57mpg Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.352s (2.352s including waiting). Image size: 235100981 bytes. 64m Normal Pulled pod/kserve-controller-manager-776cc66d47-4jgtx Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.425s (2.425s including waiting). Image size: 247676581 bytes. 64m Normal Pulled pod/model-serving-api-75bd89fdfc-t2c9k Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 1.902s (1.902s including waiting). Image size: 184925761 bytes. 64m Normal Created pod/kserve-controller-manager-776cc66d47-4jgtx Container created 64m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-f87fcf56c-dp85z_b7c7133b-16f6-4ac4-b434-c0666d0ecf64 became leader 64m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-4jgtx_1e96325a-76be-406a-b20f-6e80adf10462 became leader 64m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-57mpg_fa4753cb-7d92-482b-95d4-a744df6cf211 became leader 61m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-577f596567 from 0 to 1 61m Normal SuccessfulCreate replicaset/maas-controller-577f596567 Created pod: maas-controller-577f596567-5pmj8 61m Normal Pulling pod/maas-controller-5d77b88d59-8t5sz Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 61m Normal AddedInterface pod/maas-controller-5d77b88d59-8t5sz Add eth0 [10.132.0.44/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/maas-controller-5d77b88d59 Created pod: maas-controller-5d77b88d59-8t5sz 61m Normal AddedInterface pod/maas-controller-577f596567-5pmj8 Add eth0 [10.132.0.43/23] from ovn-kubernetes 61m Normal Pulling pod/maas-controller-577f596567-5pmj8 Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 61m Normal Created pod/maas-controller-577f596567-5pmj8 Container created 61m Normal Started pod/maas-controller-5d77b88d59-8t5sz Container started 61m Normal Created pod/maas-controller-5d77b88d59-8t5sz Container created 61m Normal Started pod/maas-controller-577f596567-5pmj8 Container started 61m Normal Pulled pod/maas-controller-5d77b88d59-8t5sz Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.691s (2.691s including waiting). Image size: 199691795 bytes. 61m Normal Pulled pod/maas-controller-577f596567-5pmj8 Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 2.762s (2.762s including waiting). Image size: 199691795 bytes. 61m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5d77b88d59-8t5sz_5c070c0b-954b-4dbe-82b0-32f757a2a50e became leader 61m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-7d94f55f4b from 0 to 1 61m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-7d94f55f4b Created pod: usage-logs-tenancy-proxy-7d94f55f4b-tw5cx 61m Normal AddedInterface pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Add eth0 [10.134.0.30/23] from ovn-kubernetes 61m Normal Pulling pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 61m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-577f596567 from 1 to 0 61m Normal SuccessfulDelete replicaset/maas-controller-577f596567 Deleted pod: maas-controller-577f596567-5pmj8 61m Normal Killing pod/maas-controller-577f596567-5pmj8 Stopping container manager 61m Normal Pulled pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 11.131s (11.131s including waiting). Image size: 1089502687 bytes. 61m Normal Created pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Container created 61m Normal Started pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Container started 60m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-9jkvx Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 60m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-9jkvx 60m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 60m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-9jkvx Add eth0 [10.133.0.39/23] from ovn-kubernetes 60m Normal Created pod/llmisvc-controller-manager-6466c554d8-9jkvx Container created 60m Normal Started pod/llmisvc-controller-manager-6466c554d8-9jkvx Container started 60m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-9jkvx Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 462ms (462ms including waiting). Image size: 235100981 bytes. 60m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-57mpg Stopping container manager 60m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-57mpg 60m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 59m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-9jkvx_5b2bc366-302a-4d99-ba89-ed651555ec78 became leader 59m Normal SuccessfulCreate replicaset/ai-gateway-controller-5d9cd99f45 Created pod: ai-gateway-controller-5d9cd99f45-lwzf6 59m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5d9cd99f45 from 0 to 1 59m Normal AddedInterface pod/ai-gateway-controller-5d9cd99f45-lwzf6 Add eth0 [10.134.0.36/23] from ovn-kubernetes 59m Normal Pulling pod/ai-gateway-controller-5d9cd99f45-lwzf6 Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" 59m Normal Started pod/ai-gateway-controller-5d9cd99f45-lwzf6 Container started 59m Normal Created pod/ai-gateway-controller-5d9cd99f45-lwzf6 Container created 59m Normal Pulled pod/ai-gateway-controller-5d9cd99f45-lwzf6 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" in 2.747s (2.747s including waiting). Image size: 183904678 bytes. 59m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5d9cd99f45-lwzf6_cc917c7d-82ed-461d-9181-e9da1acdd3e0 became leader 59m Normal SuccessfulCreate replicaset/ai-gateway-controller-77449b8677 Created pod: ai-gateway-controller-77449b8677-8jpjr 59m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-77449b8677 from 0 to 1 59m Normal Created pod/ai-gateway-controller-77449b8677-8jpjr Container created 59m Normal Pulled pod/ai-gateway-controller-77449b8677-8jpjr Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" in 324ms (324ms including waiting). Image size: 183904678 bytes. 59m Normal Pulling pod/ai-gateway-controller-77449b8677-8jpjr Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" 59m Normal Started pod/ai-gateway-controller-77449b8677-8jpjr Container started 59m Normal AddedInterface pod/ai-gateway-controller-77449b8677-8jpjr Add eth0 [10.134.0.40/23] from ovn-kubernetes 58m Normal SuccessfulDelete replicaset/ai-gateway-controller-5d9cd99f45 Deleted pod: ai-gateway-controller-5d9cd99f45-lwzf6 58m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5d9cd99f45 from 1 to 0 58m Normal Killing pod/ai-gateway-controller-5d9cd99f45-lwzf6 Stopping container manager 58m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-77449b8677-8jpjr_96c63829-3eec-42bf-ab2e-531684288039 became leader 52m Warning Unhealthy pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Liveness probe failed: Get "http://10.134.0.18:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Readiness probe failed: Get "https://10.134.0.30:8443/ready": context deadline exceeded 51m Normal AddedInterface pod/test-curl-8867-cde874 Add eth0 [10.133.0.91/23] from ovn-kubernetes 51m Normal Pulling pod/test-curl-8867-cde874 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 51m Normal Started pod/test-curl-8867-cde874 Container started 51m Normal Created pod/test-curl-8867-cde874 Container created 51m Normal Pulled pod/test-curl-8867-cde874 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.737s (2.737s including waiting). Image size: 106100986 bytes. 51m Normal Created pod/test-curl-8867-539628 Container created 51m Normal AddedInterface pod/test-curl-8867-539628 Add eth0 [10.133.0.93/23] from ovn-kubernetes 51m Normal Pulled pod/test-curl-8867-539628 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 51m Normal Started pod/test-curl-8867-539628 Container started 51m Normal Killing pod/test-curl-8867-cde874 Stopping container test-curl-8867-cde874 51m Normal Killing pod/test-curl-8867-539628 Stopping container test-curl-8867-539628 47m Normal Pulled pod/test-curl-8876-e8f1d5 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 47m Normal Created pod/test-curl-8876-e8f1d5 Container created 47m Normal AddedInterface pod/test-curl-8876-e8f1d5 Add eth0 [10.133.0.117/23] from ovn-kubernetes 47m Normal Started pod/test-curl-8876-e8f1d5 Container started 47m Normal Killing pod/test-curl-8876-e8f1d5 Stopping container test-curl-8876-e8f1d5 47m Normal Pulling pod/test-curl-8867-182852 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 47m Normal AddedInterface pod/test-curl-8867-182852 Add eth0 [10.132.0.115/23] from ovn-kubernetes 46m Normal Started pod/test-curl-8867-182852 Container started 46m Normal Created pod/test-curl-8867-182852 Container created 46m Normal Pulled pod/test-curl-8867-182852 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 17.653s (17.653s including waiting). Image size: 106100986 bytes. 46m Normal Killing pod/test-curl-8867-182852 Stopping container test-curl-8867-182852 46m Normal AddedInterface pod/test-curl-8867-4aad04 Add eth0 [10.132.0.118/23] from ovn-kubernetes 46m Normal Created pod/test-curl-8867-4aad04 Container created 46m Normal Started pod/test-curl-8867-4aad04 Container started 46m Normal Pulled pod/test-curl-8867-4aad04 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 46m Normal Started pod/test-curl-8867-280ef2 Container started 46m Normal Killing pod/test-curl-8867-4aad04 Stopping container test-curl-8867-4aad04 46m Normal Created pod/test-curl-8867-280ef2 Container created 46m Normal Pulled pod/test-curl-8867-280ef2 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/test-curl-8867-280ef2 Add eth0 [10.132.0.119/23] from ovn-kubernetes 46m Normal Killing pod/test-curl-8867-280ef2 Stopping container test-curl-8867-280ef2 46m Normal Started pod/test-curl-8867-fbfc64 Container started 46m Normal Created pod/test-curl-8867-fbfc64 Container created 46m Normal Pulled pod/test-curl-8867-fbfc64 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/test-curl-8867-fbfc64 Add eth0 [10.132.0.120/23] from ovn-kubernetes 46m Normal Killing pod/test-curl-8867-fbfc64 Stopping container test-curl-8867-fbfc64 44m Normal AddedInterface pod/test-curl-8876-02f73b Add eth0 [10.132.0.134/23] from ovn-kubernetes 44m Normal Started pod/test-curl-8876-02f73b Container started 44m Normal Created pod/test-curl-8876-02f73b Container created 44m Normal Pulled pod/test-curl-8876-02f73b Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 44m Normal Killing pod/test-curl-8876-02f73b Stopping container test-curl-8876-02f73b 23m Normal Pulled pod/test-curl-8876-8ac5bc Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/test-curl-8876-8ac5bc Add eth0 [10.132.0.223/23] from ovn-kubernetes 23m Normal Created pod/test-curl-8876-8ac5bc Container created 23m Normal Started pod/test-curl-8876-8ac5bc Container started 23m Normal Created pod/test-curl-8876-3723ff Container created 23m Normal Started pod/test-curl-8876-3723ff Container started 23m Normal AddedInterface pod/test-curl-8876-3723ff Add eth0 [10.132.0.224/23] from ovn-kubernetes 23m Normal Pulled pod/test-curl-8876-3723ff Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 23m Normal Killing pod/test-curl-8876-8ac5bc Stopping container test-curl-8876-8ac5bc 23m Normal Killing pod/test-curl-8876-3723ff Stopping container test-curl-8876-3723ff 22m Warning Unhealthy pod/opendatahub-operator-controller-manager-8484dd48c9-2v6v6 Readiness probe failed: Get "http://10.134.0.18:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 22m Warning Unhealthy pod/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx Liveness probe failed: Get "https://10.134.0.30:8443/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 21m Normal AddedInterface pod/test-curl-8864-f40e45 Add eth0 [10.132.0.240/23] from ovn-kubernetes 21m Normal Pulled pod/test-curl-8864-f40e45 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 21m Normal Started pod/test-curl-8864-f40e45 Container started 21m Normal Created pod/test-curl-8864-f40e45 Container created 21m Normal Killing pod/test-curl-8864-f40e45 Stopping container test-curl-8864-f40e45 14m Normal SuccessfulDelete replicaset/maas-controller-5d77b88d59 Deleted pod: maas-controller-5d77b88d59-8t5sz 14m Normal Killing pod/maas-controller-5d77b88d59-8t5sz Stopping container manager 13m Normal SuccessfulCreate replicaset/maas-controller-5d77b88d59 Created pod: maas-controller-5d77b88d59-2jptv 13m Normal AddedInterface pod/maas-controller-5d77b88d59-2jptv Add eth0 [10.132.1.1/23] from ovn-kubernetes 13m Normal Pulling pod/maas-controller-5d77b88d59-2jptv Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal Started pod/maas-controller-5d77b88d59-2jptv Container started 13m Normal Pulled pod/maas-controller-5d77b88d59-2jptv Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 513ms (513ms including waiting). Image size: 199691795 bytes. 13m Normal Created pod/maas-controller-5d77b88d59-2jptv Container created 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5d77b88d59-2jptv_a4b7d248-75c4-4c68-9fa8-883199199834 became leader 13m Normal SuccessfulDelete replicaset/maas-controller-5d77b88d59 Deleted pod: maas-controller-5d77b88d59-2jptv 13m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-5d77b88d59 from 1 to 0 13m Normal Killing pod/maas-controller-5d77b88d59-2jptv Stopping container manager 13m Normal AddedInterface pod/maas-controller-5d77b88d59-t8d5z Add eth0 [10.132.1.3/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/maas-controller-5d77b88d59 Created pod: maas-controller-5d77b88d59-t8d5z 13m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-5d77b88d59 from 0 to 1 13m Normal Created pod/maas-controller-5d77b88d59-t8d5z Container created 13m Normal Started pod/maas-controller-5d77b88d59-t8d5z Container started 13m Normal Pulled pod/maas-controller-5d77b88d59-t8d5z Successfully pulled image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" in 320ms (320ms including waiting). Image size: 199691795 bytes. 13m Normal Pulling pod/maas-controller-5d77b88d59-t8d5z Pulling image "quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726" 13m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5d77b88d59-t8d5z_83a9008a-9aec-44ab-9051-f43b381ffea0 became leader