=== events in opendatahub (2026-10-31T04:17:33Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 19m Normal Scheduled pod/test-curl-7279-859d10 Successfully assigned opendatahub/test-curl-7279-859d10 to ip-10-0-141-153.us-east-2.compute.internal 55m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl to ip-10-0-141-153.us-east-2.compute.internal 55m Normal Scheduled pod/model-serving-api-75bd89fdfc-s22vp Successfully assigned opendatahub/model-serving-api-75bd89fdfc-s22vp to ip-10-0-141-153.us-east-2.compute.internal 46m Normal Scheduled pod/test-curl-7264-46ccf5 Successfully assigned opendatahub/test-curl-7264-46ccf5 to ip-10-0-129-132.us-east-2.compute.internal 46m Normal Scheduled pod/test-curl-7264-189b9e Successfully assigned opendatahub/test-curl-7264-189b9e to ip-10-0-129-132.us-east-2.compute.internal 42m Normal Scheduled pod/test-curl-7261-db2154 Successfully assigned opendatahub/test-curl-7261-db2154 to ip-10-0-141-153.us-east-2.compute.internal 54m Normal Scheduled pod/maas-controller-579cf7c6d-f9vsn Successfully assigned opendatahub/maas-controller-579cf7c6d-f9vsn to ip-10-0-142-76.us-east-2.compute.internal 54m Normal Scheduled pod/maas-controller-586ddd746f-nv8gm Successfully assigned opendatahub/maas-controller-586ddd746f-nv8gm to ip-10-0-142-76.us-east-2.compute.internal 41m Normal Scheduled pod/test-curl-7261-004b66 Successfully assigned opendatahub/test-curl-7261-004b66 to ip-10-0-141-153.us-east-2.compute.internal 52m Normal Scheduled pod/ai-gateway-controller-6fd9f75659-4s8nf Successfully assigned opendatahub/ai-gateway-controller-6fd9f75659-4s8nf to ip-10-0-129-132.us-east-2.compute.internal 54m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-ntrp9 Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9 to ip-10-0-141-153.us-east-2.compute.internal 16m Normal Scheduled pod/maas-controller-586ddd746f-ftqn6 Successfully assigned opendatahub/maas-controller-586ddd746f-ftqn6 to ip-10-0-142-76.us-east-2.compute.internal 52m Normal Scheduled pod/ai-gateway-controller-5d9cd99f45-gpzfj Successfully assigned opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj to ip-10-0-142-76.us-east-2.compute.internal 56m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z to ip-10-0-142-76.us-east-2.compute.internal 16m Normal Scheduled pod/maas-controller-586ddd746f-hdwdt Successfully assigned opendatahub/maas-controller-586ddd746f-hdwdt to ip-10-0-142-76.us-east-2.compute.internal 55m Normal Scheduled pod/odh-model-controller-769875db56-6r4mx Successfully assigned opendatahub/odh-model-controller-769875db56-6r4mx to ip-10-0-142-76.us-east-2.compute.internal 43m Normal Scheduled pod/test-curl-7267-06eeca Successfully assigned opendatahub/test-curl-7267-06eeca to ip-10-0-141-153.us-east-2.compute.internal 43m Normal Scheduled pod/test-curl-7267-c01a17 Successfully assigned opendatahub/test-curl-7267-c01a17 to ip-10-0-141-153.us-east-2.compute.internal 22m Normal Scheduled pod/odh-model-controller-769875db56-s9sn6 Successfully assigned opendatahub/odh-model-controller-769875db56-s9sn6 to ip-10-0-142-76.us-east-2.compute.internal 43m Normal Scheduled pod/test-curl-7267-0a2304 Successfully assigned opendatahub/test-curl-7267-0a2304 to ip-10-0-141-153.us-east-2.compute.internal 54m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb to ip-10-0-142-76.us-east-2.compute.internal 55m Normal Scheduled pod/kserve-controller-manager-776cc66d47-wnnv7 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-wnnv7 to ip-10-0-142-76.us-east-2.compute.internal 43m Normal Scheduled pod/test-curl-7267-a68cf2 Successfully assigned opendatahub/test-curl-7267-a68cf2 to ip-10-0-141-153.us-east-2.compute.internal 56m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Add eth0 [10.133.0.28/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-wlb6z 56m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 56m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 56m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.249s (2.249s including waiting). Image size: 195729000 bytes. 56m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-wlb6z_f23ec5d8-585f-4ed2-8d84-6229b25221ea became leader 55m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-s22vp 55m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 55m Warning FailedMount pod/model-serving-api-75bd89fdfc-s22vp MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 55m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 55m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-6r4mx 55m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-wnnv7 55m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-wd8tl 55m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 55m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 55m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Add eth0 [10.134.0.26/23] from ovn-kubernetes 55m Normal AddedInterface pod/odh-model-controller-769875db56-6r4mx Add eth0 [10.133.0.32/23] from ovn-kubernetes 55m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 55m Normal Pulling pod/model-serving-api-75bd89fdfc-s22vp Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 55m Normal AddedInterface pod/model-serving-api-75bd89fdfc-s22vp Add eth0 [10.134.0.27/23] from ovn-kubernetes 55m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-wnnv7 Add eth0 [10.133.0.31/23] from ovn-kubernetes 55m Normal Pulled pod/kserve-controller-manager-776cc66d47-wnnv7 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.582s (2.582s including waiting). Image size: 247676581 bytes. 55m Normal Started pod/model-serving-api-75bd89fdfc-s22vp Container started 55m Normal Created pod/model-serving-api-75bd89fdfc-s22vp Container created 55m Normal Pulled pod/model-serving-api-75bd89fdfc-s22vp Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 1.837s (1.837s including waiting). Image size: 184925761 bytes. 55m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-wnnv7_a732a359-8d5a-4208-8988-2341c7d90e60 became leader 55m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Container created 55m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Container started 55m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-6r4mx_c3de937b-2fff-4e0c-8bb5-8bedbd95188a became leader 55m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.075s (3.075s including waiting). Image size: 235100981 bytes. 55m Normal Pulled pod/odh-model-controller-769875db56-6r4mx Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.03s (3.03s including waiting). Image size: 236755567 bytes. 55m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-wd8tl_dd9fc07b-3d41-4839-b5dd-76b6336e9e5c became leader 54m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 54m Normal SuccessfulCreate replicaset/maas-controller-586ddd746f Created pod: maas-controller-586ddd746f-nv8gm 54m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-f9vsn 54m Normal AddedInterface pod/maas-controller-579cf7c6d-f9vsn Add eth0 [10.133.0.34/23] from ovn-kubernetes 54m Normal Pulling pod/maas-controller-579cf7c6d-f9vsn Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 54m Normal AddedInterface pod/maas-controller-586ddd746f-nv8gm Add eth0 [10.133.0.35/23] from ovn-kubernetes 54m Normal Created pod/maas-controller-579cf7c6d-f9vsn Container created 54m Normal Pulled pod/maas-controller-579cf7c6d-f9vsn Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 3.016s (3.016s including waiting). Image size: 200053271 bytes. 54m Normal Started pod/maas-controller-579cf7c6d-f9vsn Container started 54m Normal Pulled pod/maas-controller-586ddd746f-nv8gm Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.889s (2.889s including waiting). Image size: 200053271 bytes. 54m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-579cf7c6d-f9vsn_96247bd4-6538-4885-b2a8-a5d17c367c01 became leader 54m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 54m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 54m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Add eth0 [10.133.0.36/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-68drb 54m Normal Killing pod/maas-controller-579cf7c6d-f9vsn Stopping container manager 54m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 54m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-f9vsn 54m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 12.115s (12.115s including waiting). Image size: 1089502687 bytes. 54m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-586ddd746f-nv8gm_7c1dc6fd-d326-43c0-abb2-26245bc860dc became leader 54m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 54m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-ntrp9 54m Normal Created pod/llmisvc-controller-manager-6466c554d8-ntrp9 Container created 54m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-ntrp9 Add eth0 [10.134.0.30/23] from ovn-kubernetes 54m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-ntrp9 Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 54m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-ntrp9 Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 413ms (413ms including waiting). Image size: 235100981 bytes. 54m Normal Started pod/llmisvc-controller-manager-6466c554d8-ntrp9 Container started 53m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-wd8tl Stopping container manager 53m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 53m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-wd8tl 53m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-ntrp9_bcc8bed3-bb4e-4c00-a081-519a08727d99 became leader 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5d9cd99f45 from 0 to 1 52m Normal SuccessfulCreate replicaset/ai-gateway-controller-5d9cd99f45 Created pod: ai-gateway-controller-5d9cd99f45-gpzfj 52m Normal AddedInterface pod/ai-gateway-controller-5d9cd99f45-gpzfj Add eth0 [10.133.0.47/23] from ovn-kubernetes 52m Normal Pulling pod/ai-gateway-controller-5d9cd99f45-gpzfj Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" 52m Normal Started pod/ai-gateway-controller-5d9cd99f45-gpzfj Container started 52m Normal Pulled pod/ai-gateway-controller-5d9cd99f45-gpzfj Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" in 3.895s (3.895s including waiting). Image size: 184085929 bytes. 52m Normal Created pod/ai-gateway-controller-5d9cd99f45-gpzfj Container created 52m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5d9cd99f45-gpzfj_e9cbde16-8914-47aa-945f-88d470730ebf became leader 52m Normal SuccessfulCreate replicaset/ai-gateway-controller-6fd9f75659 Created pod: ai-gateway-controller-6fd9f75659-4s8nf 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-6fd9f75659 from 0 to 1 52m Normal Pulling pod/ai-gateway-controller-6fd9f75659-4s8nf Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" 52m Normal AddedInterface pod/ai-gateway-controller-6fd9f75659-4s8nf Add eth0 [10.132.0.53/23] from ovn-kubernetes 52m Normal Created pod/ai-gateway-controller-6fd9f75659-4s8nf Container created 52m Normal Pulled pod/ai-gateway-controller-6fd9f75659-4s8nf Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-191" in 2.901s (2.901s including waiting). Image size: 184085929 bytes. 52m Normal Started pod/ai-gateway-controller-6fd9f75659-4s8nf Container started 52m Normal SuccessfulDelete replicaset/ai-gateway-controller-5d9cd99f45 Deleted pod: ai-gateway-controller-5d9cd99f45-gpzfj 52m Normal Killing pod/ai-gateway-controller-5d9cd99f45-gpzfj Stopping container manager 52m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-5d9cd99f45 from 1 to 0 52m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-6fd9f75659-4s8nf_cf9c3a94-6beb-47ab-bcbb-89c7d016fba8 became leader 46m Normal Pulling pod/test-curl-7264-46ccf5 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 46m Normal AddedInterface pod/test-curl-7264-46ccf5 Add eth0 [10.132.0.139/23] from ovn-kubernetes 46m Normal Started pod/test-curl-7264-46ccf5 Container started 46m Normal Created pod/test-curl-7264-46ccf5 Container created 46m Normal Pulled pod/test-curl-7264-46ccf5 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 3.99s (3.99s including waiting). Image size: 106100986 bytes. 46m Normal AddedInterface pod/test-curl-7264-189b9e Add eth0 [10.132.0.142/23] from ovn-kubernetes 46m Normal Pulled pod/test-curl-7264-189b9e Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 46m Normal Killing pod/test-curl-7264-46ccf5 Stopping container test-curl-7264-46ccf5 46m Normal Created pod/test-curl-7264-189b9e Container created 46m Normal Started pod/test-curl-7264-189b9e Container started 46m Normal Killing pod/test-curl-7264-189b9e Stopping container test-curl-7264-189b9e 43m Normal AddedInterface pod/test-curl-7267-0a2304 Add eth0 [10.134.0.104/23] from ovn-kubernetes 43m Normal Pulling pod/test-curl-7267-0a2304 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 43m Normal Started pod/test-curl-7267-0a2304 Container started 43m Normal Pulled pod/test-curl-7267-0a2304 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 3.836s (3.836s including waiting). Image size: 106100986 bytes. 43m Normal Created pod/test-curl-7267-0a2304 Container created 43m Normal Started pod/test-curl-7267-06eeca Container started 43m Normal Killing pod/test-curl-7267-0a2304 Stopping container test-curl-7267-0a2304 43m Normal AddedInterface pod/test-curl-7267-06eeca Add eth0 [10.134.0.105/23] from ovn-kubernetes 43m Normal Pulled pod/test-curl-7267-06eeca Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 43m Normal Created pod/test-curl-7267-06eeca Container created 43m Normal Killing pod/test-curl-7267-06eeca Stopping container test-curl-7267-06eeca 43m Normal AddedInterface pod/test-curl-7267-c01a17 Add eth0 [10.134.0.106/23] from ovn-kubernetes 43m Normal Pulled pod/test-curl-7267-c01a17 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 43m Normal Created pod/test-curl-7267-c01a17 Container created 43m Normal Started pod/test-curl-7267-c01a17 Container started 43m Normal Started pod/test-curl-7267-a68cf2 Container started 43m Normal Created pod/test-curl-7267-a68cf2 Container created 43m Normal Pulled pod/test-curl-7267-a68cf2 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/test-curl-7267-a68cf2 Add eth0 [10.134.0.107/23] from ovn-kubernetes 43m Normal Killing pod/test-curl-7267-c01a17 Stopping container test-curl-7267-c01a17 43m Normal Killing pod/test-curl-7267-a68cf2 Stopping container test-curl-7267-a68cf2 42m Normal Started pod/test-curl-7261-db2154 Container started 42m Normal Created pod/test-curl-7261-db2154 Container created 42m Normal AddedInterface pod/test-curl-7261-db2154 Add eth0 [10.134.0.109/23] from ovn-kubernetes 42m Normal Pulled pod/test-curl-7261-db2154 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 42m Normal Killing pod/test-curl-7261-db2154 Stopping container test-curl-7261-db2154 41m Normal AddedInterface pod/test-curl-7261-004b66 Add eth0 [10.134.0.120/23] from ovn-kubernetes 41m Normal Pulled pod/test-curl-7261-004b66 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 41m Normal Created pod/test-curl-7261-004b66 Container created 41m Normal Started pod/test-curl-7261-004b66 Container started 41m Normal Killing pod/test-curl-7261-004b66 Stopping container test-curl-7261-004b66 34m Normal Pulling pod/odh-model-controller-769875db56-6r4mx Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 34m Normal Pulled pod/odh-model-controller-769875db56-6r4mx Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 4.184s (4.184s including waiting). Image size: 236755567 bytes. 34m Normal Started pod/odh-model-controller-769875db56-6r4mx Container started 34m Normal Created pod/odh-model-controller-769875db56-6r4mx Container created 33m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-6r4mx_5297fffc-6c74-43a1-9ddd-689b21d56079 became leader 26m Warning Unhealthy pod/odh-model-controller-769875db56-6r4mx Liveness probe failed: Get "http://10.133.0.32:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Liveness probe failed: Get "http://10.133.0.28:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Readiness probe failed: Get "http://10.133.0.28:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-wlb6z_f23ec5d8-585f-4ed2-8d84-6229b25221ea stopped leading 25m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Readiness probe failed: Get "https://10.133.0.36:8443/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 25m Warning NodeNotReady pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Node is not ready 25m Warning NodeNotReady pod/odh-model-controller-769875db56-6r4mx Node is not ready 25m Warning NodeNotReady pod/kserve-controller-manager-776cc66d47-wnnv7 Node is not ready 24m Warning Unhealthy pod/maas-controller-586ddd746f-nv8gm Liveness probe failed: Get "http://10.133.0.35:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal Killing pod/odh-model-controller-769875db56-6r4mx Stopping container manager 24m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Liveness probe failed: Get "https://10.133.0.36:8443/health": context deadline exceeded 24m Warning Unhealthy pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Liveness probe failed: Get "http://10.133.0.28:8081/healthz": dial tcp 10.133.0.28:8081: connect: connection refused 24m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Liveness probe failed: Get "https://10.133.0.36:8443/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 24m Warning Unhealthy pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Readiness probe failed: Get "http://10.133.0.28:8081/readyz": dial tcp 10.133.0.28:8081: connect: connection refused 24m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-wnnv7 Liveness probe failed: Get "http://10.133.0.31:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal Killing pod/kserve-controller-manager-776cc66d47-wnnv7 Container manager failed liveness probe, will be restarted 24m Normal Killing pod/maas-controller-586ddd746f-nv8gm Container manager failed liveness probe, will be restarted 24m Normal Killing pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Container proxy failed liveness probe, will be restarted 24m Warning Unhealthy pod/odh-model-controller-769875db56-6r4mx Readiness probe failed: Get "http://10.133.0.32:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Readiness probe failed: Get "https://10.133.0.36:8443/ready": context deadline exceeded 24m Normal TaintManagerEviction pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Cancelling deletion of Pod opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb 24m Normal TaintManagerEviction pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Cancelling deletion of Pod opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z 24m Normal TaintManagerEviction pod/kserve-controller-manager-776cc66d47-wnnv7 Cancelling deletion of Pod opendatahub/kserve-controller-manager-776cc66d47-wnnv7 24m Normal TaintManagerEviction pod/maas-controller-586ddd746f-nv8gm Cancelling deletion of Pod opendatahub/maas-controller-586ddd746f-nv8gm 24m Normal TaintManagerEviction pod/odh-model-controller-769875db56-6r4mx Cancelling deletion of Pod opendatahub/odh-model-controller-769875db56-6r4mx 23m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Container image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" already present on machine and can be accessed by the pod 23m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-wnnv7 Readiness probe failed: Get "http://10.133.0.31:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 23m Warning Unhealthy pod/odh-model-controller-769875db56-6r4mx Readiness probe failed: Get "http://10.133.0.32:8081/readyz": dial tcp 10.133.0.32:8081: connect: connection refused 23m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Readiness probe failed: Get "https://10.133.0.36:8443/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 23m Warning Unhealthy pod/maas-controller-586ddd746f-nv8gm Readiness probe failed: Get "http://10.133.0.35:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 23m Normal Pulling pod/kserve-controller-manager-776cc66d47-wnnv7 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 23m Normal Pulling pod/maas-controller-586ddd746f-nv8gm Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 23m Warning Unhealthy pod/maas-controller-586ddd746f-nv8gm Readiness probe failed: Get "http://10.133.0.35:8081/readyz": dial tcp 10.133.0.35:8081: connect: connection refused 23m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Container created 23m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-wlb6z Container started 23m Normal Pulled pod/kserve-controller-manager-776cc66d47-wnnv7 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 4.447s (4.447s including waiting). Image size: 247676581 bytes. 23m Normal Pulled pod/maas-controller-586ddd746f-nv8gm Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.913s (2.913s including waiting). Image size: 200053271 bytes. 23m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-wnnv7 Readiness probe failed: Get "http://10.133.0.31:8081/readyz": dial tcp 10.133.0.31:8081: connect: connection refused 23m Normal Created pod/maas-controller-586ddd746f-nv8gm Container created 23m Normal Started pod/maas-controller-586ddd746f-nv8gm Container started 23m Normal Started pod/kserve-controller-manager-776cc66d47-wnnv7 Container started 23m Normal Created pod/kserve-controller-manager-776cc66d47-wnnv7 Container created 23m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Container image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" already present on machine and can be accessed by the pod 23m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Container started 23m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-68drb Container created 23m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-wnnv7_382362e7-fb94-4e6d-9c70-deb23def029a became leader 23m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-586ddd746f-nv8gm_d6c61e38-e38b-4a8a-b115-151a7e28c9c9 became leader 23m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-wlb6z_6fc08977-8712-46fd-9ce3-5c58f46b5baa became leader 22m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-s9sn6 22m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 22m Normal AddedInterface pod/odh-model-controller-769875db56-s9sn6 Add eth0 [10.133.0.142/23] from ovn-kubernetes 22m Normal Created pod/odh-model-controller-769875db56-s9sn6 Container created 22m Normal Started pod/odh-model-controller-769875db56-s9sn6 Container started 22m Normal Pulled pod/odh-model-controller-769875db56-s9sn6 Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 566ms (566ms including waiting). Image size: 236755567 bytes. 22m Normal Pulling pod/odh-model-controller-769875db56-s9sn6 Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 22m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-s9sn6_2db66a4a-c46f-4e9b-8169-b22ace54a56c became leader 19m Normal Started pod/test-curl-7279-859d10 Container started 19m Normal AddedInterface pod/test-curl-7279-859d10 Add eth0 [10.134.0.227/23] from ovn-kubernetes 19m Normal Pulled pod/test-curl-7279-859d10 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 19m Normal Created pod/test-curl-7279-859d10 Container created 19m Normal Killing pod/test-curl-7279-859d10 Stopping container test-curl-7279-859d10 16m Normal Killing pod/maas-controller-586ddd746f-nv8gm Stopping container manager 16m Normal SuccessfulDelete replicaset/maas-controller-586ddd746f Deleted pod: maas-controller-586ddd746f-nv8gm 16m Normal SuccessfulCreate replicaset/maas-controller-586ddd746f Created pod: maas-controller-586ddd746f-ftqn6 16m Normal Pulled pod/maas-controller-586ddd746f-ftqn6 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 569ms (569ms including waiting). Image size: 200053271 bytes. 16m Normal AddedInterface pod/maas-controller-586ddd746f-ftqn6 Add eth0 [10.133.0.150/23] from ovn-kubernetes 16m Normal Pulling pod/maas-controller-586ddd746f-ftqn6 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 16m Normal Started pod/maas-controller-586ddd746f-ftqn6 Container started 16m Normal Created pod/maas-controller-586ddd746f-ftqn6 Container created 16m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-586ddd746f-ftqn6_b34064d5-b384-4df1-b29b-171b8c6b72c9 became leader 16m Normal SuccessfulDelete replicaset/maas-controller-586ddd746f Deleted pod: maas-controller-586ddd746f-ftqn6 16m Normal Killing pod/maas-controller-586ddd746f-ftqn6 Stopping container manager 16m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-586ddd746f from 1 to 0 16m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-586ddd746f from 0 to 1 16m Normal SuccessfulCreate replicaset/maas-controller-586ddd746f Created pod: maas-controller-586ddd746f-hdwdt 16m Normal AddedInterface pod/maas-controller-586ddd746f-hdwdt Add eth0 [10.133.0.152/23] from ovn-kubernetes 16m Normal Pulling pod/maas-controller-586ddd746f-hdwdt Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 16m Normal Started pod/maas-controller-586ddd746f-hdwdt Container started 16m Normal Created pod/maas-controller-586ddd746f-hdwdt Container created 16m Normal Pulled pod/maas-controller-586ddd746f-hdwdt Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 428ms (428ms including waiting). Image size: 200053271 bytes. 16m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-586ddd746f-hdwdt_8e20a0d0-9de3-41fb-9719-7fc2521c95dc became leader