=== events in opendatahub (2026-11-04T01:52:16Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 34m Normal Scheduled pod/test-curl-7005-0af584 Successfully assigned opendatahub/test-curl-7005-0af584 to ip-10-0-132-67.us-east-2.compute.internal 40m Normal Scheduled pod/ai-gateway-controller-568986c44-m79cq Successfully assigned opendatahub/ai-gateway-controller-568986c44-m79cq to ip-10-0-136-96.us-east-2.compute.internal 17m Normal Scheduled pod/maas-controller-5bf8c4d6cc-brrh6 Successfully assigned opendatahub/maas-controller-5bf8c4d6cc-brrh6 to ip-10-0-136-96.us-east-2.compute.internal 34m Normal Scheduled pod/test-curl-7005-ffcb5a Successfully assigned opendatahub/test-curl-7005-ffcb5a to ip-10-0-132-67.us-east-2.compute.internal 43m Normal Scheduled pod/maas-controller-5bf8c4d6cc-v9np2 Successfully assigned opendatahub/maas-controller-5bf8c4d6cc-v9np2 to ip-10-0-140-186.us-east-2.compute.internal 43m Normal Scheduled pod/model-serving-api-75bd89fdfc-cgt26 Successfully assigned opendatahub/model-serving-api-75bd89fdfc-cgt26 to ip-10-0-136-96.us-east-2.compute.internal 31m Normal Scheduled pod/test-curl-6987-a1d191 Successfully assigned opendatahub/test-curl-6987-a1d191 to ip-10-0-132-67.us-east-2.compute.internal 40m Normal Scheduled pod/ai-gateway-controller-7cb8f6bb64-q67lm Successfully assigned opendatahub/ai-gateway-controller-7cb8f6bb64-q67lm to ip-10-0-136-96.us-east-2.compute.internal 42m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-ntcsr Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-ntcsr to ip-10-0-132-67.us-east-2.compute.internal 43m Normal Scheduled pod/odh-model-controller-769875db56-jr85b Successfully assigned opendatahub/odh-model-controller-769875db56-jr85b to ip-10-0-140-186.us-east-2.compute.internal 43m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-bzqjr to ip-10-0-132-67.us-east-2.compute.internal 43m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw to ip-10-0-140-186.us-east-2.compute.internal 43m Normal Scheduled pod/kserve-controller-manager-776cc66d47-mvf9p Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-mvf9p to ip-10-0-140-186.us-east-2.compute.internal 31m Normal Scheduled pod/test-curl-6987-cd29e9 Successfully assigned opendatahub/test-curl-6987-cd29e9 to ip-10-0-132-67.us-east-2.compute.internal 17m Normal Scheduled pod/maas-controller-5bf8c4d6cc-vmh4f Successfully assigned opendatahub/maas-controller-5bf8c4d6cc-vmh4f to ip-10-0-136-96.us-east-2.compute.internal 43m Normal Scheduled pod/maas-controller-8b95c9c66-v64v2 Successfully assigned opendatahub/maas-controller-8b95c9c66-v64v2 to ip-10-0-140-186.us-east-2.compute.internal 31m Normal Scheduled pod/test-curl-6987-f1def8 Successfully assigned opendatahub/test-curl-6987-f1def8 to ip-10-0-132-67.us-east-2.compute.internal 31m Normal Scheduled pod/test-curl-6987-5dc600 Successfully assigned opendatahub/test-curl-6987-5dc600 to ip-10-0-132-67.us-east-2.compute.internal 44m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-q9rwc to ip-10-0-132-67.us-east-2.compute.internal 44m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-q9rwc 44m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 44m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Add eth0 [10.132.0.43/23] from ovn-kubernetes 44m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 44m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.759s (2.759s including waiting). Image size: 195729000 bytes. 44m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Container created 44m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-q9rwc Container started 44m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-q9rwc_2bd26c35-0113-4aa3-9384-f82a214e2855 became leader 43m Normal AddedInterface pod/maas-controller-8b95c9c66-v64v2 Add eth0 [10.134.0.30/23] from ovn-kubernetes 43m Normal Pulling pod/maas-controller-8b95c9c66-v64v2 Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 43m Normal SuccessfulCreate replicaset/maas-controller-8b95c9c66 Created pod: maas-controller-8b95c9c66-v64v2 43m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-8b95c9c66 from 0 to 1 43m Normal SuccessfulCreate replicaset/maas-controller-5bf8c4d6cc Created pod: maas-controller-5bf8c4d6cc-v9np2 43m Normal AddedInterface pod/maas-controller-5bf8c4d6cc-v9np2 Add eth0 [10.134.0.31/23] from ovn-kubernetes 43m Normal Started pod/maas-controller-8b95c9c66-v64v2 Container started 43m Normal Pulled pod/maas-controller-8b95c9c66-v64v2 Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 4.843s (4.843s including waiting). Image size: 200103442 bytes. 43m Normal Pulled pod/maas-controller-5bf8c4d6cc-v9np2 Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 4.489s (4.489s including waiting). Image size: 200103442 bytes. 43m Normal Created pod/maas-controller-8b95c9c66-v64v2 Container created 43m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-9s6bw 43m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-8b95c9c66-v64v2_9a459069-f838-44ac-a0f0-a03ebeb03f23 became leader 43m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 43m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Add eth0 [10.134.0.32/23] from ovn-kubernetes 43m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 43m Normal Pulling pod/model-serving-api-75bd89fdfc-cgt26 Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 43m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-mvf9p 43m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-mvf9p Add eth0 [10.134.0.33/23] from ovn-kubernetes 43m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 43m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Add eth0 [10.132.0.46/23] from ovn-kubernetes 43m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 43m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 43m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 43m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-cgt26 43m Normal AddedInterface pod/model-serving-api-75bd89fdfc-cgt26 Add eth0 [10.133.0.23/23] from ovn-kubernetes 43m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-jr85b 43m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-bzqjr 43m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 43m Normal AddedInterface pod/odh-model-controller-769875db56-jr85b Add eth0 [10.134.0.34/23] from ovn-kubernetes 43m Normal Pulled pod/model-serving-api-75bd89fdfc-cgt26 Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.006s (2.006s including waiting). Image size: 184925761 bytes. 43m Normal Created pod/model-serving-api-75bd89fdfc-cgt26 Container created 43m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Container started 43m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.251s (2.251s including waiting). Image size: 235100981 bytes. 43m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Container created 43m Normal Started pod/model-serving-api-75bd89fdfc-cgt26 Container started 43m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-bzqjr_acc57eda-27cb-41f6-b4ae-c11e650dace0 became leader 43m Normal Pulled pod/kserve-controller-manager-776cc66d47-mvf9p Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.729s (3.729s including waiting). Image size: 247676581 bytes. 43m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-mvf9p_36dcbba5-dd2c-4cfb-a6a6-50628a0cdb53 became leader 43m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-jr85b_4de66748-79a6-4aeb-927c-e9b20569f7af became leader 43m Normal Pulled pod/odh-model-controller-769875db56-jr85b Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 4.339s (4.339s including waiting). Image size: 236755567 bytes. 43m Normal Killing pod/maas-controller-8b95c9c66-v64v2 Stopping container manager 43m Normal SuccessfulDelete replicaset/maas-controller-8b95c9c66 Deleted pod: maas-controller-8b95c9c66-v64v2 43m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-8b95c9c66 from 1 to 0 43m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 13.536s (13.536s including waiting). Image size: 1089502687 bytes. 43m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5bf8c4d6cc-v9np2_9c2a12ee-389f-4362-b1fc-07b4c41a9288 became leader 42m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 42m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-ntcsr 42m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-ntcsr Add eth0 [10.132.0.48/23] from ovn-kubernetes 42m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-ntcsr Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 42m Normal Started pod/llmisvc-controller-manager-6466c554d8-ntcsr Container started 42m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-ntcsr Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 589ms (589ms including waiting). Image size: 235100981 bytes. 42m Normal Created pod/llmisvc-controller-manager-6466c554d8-ntcsr Container created 42m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 42m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-bzqjr 42m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-bzqjr Stopping container manager 41m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-ntcsr_07b79036-7a7d-4e5d-ba5f-7916495e8d6b became leader 41m Normal Pulled pod/maas-controller-5bf8c4d6cc-v9np2 Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 542ms (542ms including waiting). Image size: 200103442 bytes. 40m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5bf8c4d6cc-v9np2_1bbd33e4-964b-4256-9361-ad0a0e451784 became leader 40m Normal Pulling pod/ai-gateway-controller-7cb8f6bb64-q67lm Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" 40m Normal SuccessfulCreate replicaset/ai-gateway-controller-7cb8f6bb64 Created pod: ai-gateway-controller-7cb8f6bb64-q67lm 40m Normal AddedInterface pod/ai-gateway-controller-7cb8f6bb64-q67lm Add eth0 [10.133.0.36/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-7cb8f6bb64 from 0 to 1 40m Normal Created pod/ai-gateway-controller-7cb8f6bb64-q67lm Container created 40m Normal Started pod/ai-gateway-controller-7cb8f6bb64-q67lm Container started 40m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-7cb8f6bb64-q67lm_e9368945-8699-434e-9f3f-4dd35d2d4247 became leader 40m Normal Pulled pod/ai-gateway-controller-7cb8f6bb64-q67lm Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" in 4.514s (4.514s including waiting). Image size: 184136102 bytes. 40m Normal SuccessfulCreate replicaset/ai-gateway-controller-568986c44 Created pod: ai-gateway-controller-568986c44-m79cq 40m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-568986c44 from 0 to 1 40m Normal Pulled pod/ai-gateway-controller-568986c44-m79cq Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" in 466ms (466ms including waiting). Image size: 184136102 bytes. 40m Normal Started pod/ai-gateway-controller-568986c44-m79cq Container started 40m Normal Created pod/ai-gateway-controller-568986c44-m79cq Container created 40m Normal Pulling pod/ai-gateway-controller-568986c44-m79cq Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-stable" 40m Normal AddedInterface pod/ai-gateway-controller-568986c44-m79cq Add eth0 [10.133.0.40/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-7cb8f6bb64 from 1 to 0 40m Normal Killing pod/ai-gateway-controller-7cb8f6bb64-q67lm Stopping container manager 40m Normal SuccessfulDelete replicaset/ai-gateway-controller-7cb8f6bb64 Deleted pod: ai-gateway-controller-7cb8f6bb64-q67lm 40m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-568986c44-m79cq_219576b2-5952-47d4-a741-5511b0b73581 became leader 34m Normal AddedInterface pod/test-curl-7005-0af584 Add eth0 [10.132.0.92/23] from ovn-kubernetes 34m Normal Pulling pod/test-curl-7005-0af584 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 34m Normal Started pod/test-curl-7005-0af584 Container started 34m Normal Pulled pod/test-curl-7005-0af584 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 9.997s (9.997s including waiting). Image size: 106100986 bytes. 34m Normal Created pod/test-curl-7005-0af584 Container created 34m Normal AddedInterface pod/test-curl-7005-ffcb5a Add eth0 [10.132.0.93/23] from ovn-kubernetes 34m Normal Started pod/test-curl-7005-ffcb5a Container started 34m Normal Created pod/test-curl-7005-ffcb5a Container created 34m Normal Killing pod/test-curl-7005-0af584 Stopping container test-curl-7005-0af584 34m Normal Pulled pod/test-curl-7005-ffcb5a Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 34m Normal Killing pod/test-curl-7005-ffcb5a Stopping container test-curl-7005-ffcb5a 31m Normal Pulled pod/test-curl-6987-a1d191 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/test-curl-6987-a1d191 Add eth0 [10.132.0.107/23] from ovn-kubernetes 31m Normal Created pod/test-curl-6987-a1d191 Container created 31m Normal Started pod/test-curl-6987-a1d191 Container started 31m Normal Created pod/test-curl-6987-cd29e9 Container created 31m Normal Started pod/test-curl-6987-cd29e9 Container started 31m Normal Pulled pod/test-curl-6987-cd29e9 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/test-curl-6987-cd29e9 Add eth0 [10.132.0.109/23] from ovn-kubernetes 31m Normal Killing pod/test-curl-6987-a1d191 Stopping container test-curl-6987-a1d191 31m Normal AddedInterface pod/test-curl-6987-5dc600 Add eth0 [10.132.0.110/23] from ovn-kubernetes 31m Normal Pulled pod/test-curl-6987-5dc600 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 31m Normal Killing pod/test-curl-6987-cd29e9 Stopping container test-curl-6987-cd29e9 31m Normal Created pod/test-curl-6987-5dc600 Container created 31m Normal Started pod/test-curl-6987-5dc600 Container started 31m Normal Started pod/test-curl-6987-f1def8 Container started 31m Normal AddedInterface pod/test-curl-6987-f1def8 Add eth0 [10.132.0.111/23] from ovn-kubernetes 31m Normal Pulled pod/test-curl-6987-f1def8 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 31m Normal Created pod/test-curl-6987-f1def8 Container created 31m Normal Killing pod/test-curl-6987-5dc600 Stopping container test-curl-6987-5dc600 31m Normal Killing pod/test-curl-6987-f1def8 Stopping container test-curl-6987-f1def8 26m Warning Unhealthy pod/odh-model-controller-769875db56-jr85b Liveness probe failed: Get "http://10.134.0.34:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Liveness probe failed: Get "https://10.134.0.32:8443/health": context deadline exceeded 26m Warning Unhealthy pod/maas-controller-5bf8c4d6cc-v9np2 Liveness probe failed: Get "http://10.134.0.31:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal Killing pod/odh-model-controller-769875db56-jr85b Container manager failed liveness probe, will be restarted 26m Normal Killing pod/maas-controller-5bf8c4d6cc-v9np2 Container manager failed liveness probe, will be restarted 26m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Readiness probe failed: Get "https://10.134.0.32:8443/ready": context deadline exceeded 26m Warning Unhealthy pod/odh-model-controller-769875db56-jr85b Readiness probe failed: Get "http://10.134.0.34:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/maas-controller-5bf8c4d6cc-v9np2 Readiness probe failed: Get "http://10.134.0.31:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-mvf9p Readiness probe failed: Get "http://10.134.0.33:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-mvf9p Liveness probe failed: Get "http://10.134.0.33:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Liveness probe failed: Get "https://10.134.0.32:8443/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 26m Normal Killing pod/kserve-controller-manager-776cc66d47-mvf9p Container manager failed liveness probe, will be restarted 26m Normal Killing pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Container proxy failed liveness probe, will be restarted 26m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-mvf9p Readiness probe failed: HTTP probe failed with statuscode: 500 26m Warning Unhealthy pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Readiness probe failed: Get "https://10.134.0.32:8443/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 25m Normal TaintManagerEviction pod/odh-model-controller-769875db56-jr85b Cancelling deletion of Pod opendatahub/odh-model-controller-769875db56-jr85b 25m Normal Pulling pod/odh-model-controller-769875db56-jr85b Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 25m Normal TaintManagerEviction pod/kserve-controller-manager-776cc66d47-mvf9p Cancelling deletion of Pod opendatahub/kserve-controller-manager-776cc66d47-mvf9p 25m Normal TaintManagerEviction pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Cancelling deletion of Pod opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw 25m Normal Pulling pod/maas-controller-5bf8c4d6cc-v9np2 Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 25m Normal TaintManagerEviction pod/maas-controller-5bf8c4d6cc-v9np2 Cancelling deletion of Pod opendatahub/maas-controller-5bf8c4d6cc-v9np2 25m Normal Pulling pod/kserve-controller-manager-776cc66d47-mvf9p Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 25m Normal Pulled pod/kserve-controller-manager-776cc66d47-mvf9p Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 8.666s (8.666s including waiting). Image size: 247676581 bytes. 25m Warning Unhealthy pod/odh-model-controller-769875db56-jr85b Readiness probe failed: Get "http://10.134.0.34:8081/readyz": dial tcp 10.134.0.34:8081: connect: connection refused 25m Normal Pulled pod/maas-controller-5bf8c4d6cc-v9np2 Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 9.067s (9.067s including waiting). Image size: 200103442 bytes. 25m Warning Unhealthy pod/kserve-controller-manager-776cc66d47-mvf9p Readiness probe failed: Get "http://10.134.0.33:8081/readyz": dial tcp 10.134.0.33:8081: connect: connection refused 25m Normal Pulled pod/odh-model-controller-769875db56-jr85b Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 9.089s (9.089s including waiting). Image size: 236755567 bytes. 25m Warning Unhealthy pod/maas-controller-5bf8c4d6cc-v9np2 Readiness probe failed: Get "http://10.134.0.31:8081/readyz": dial tcp 10.134.0.31:8081: connect: connection refused 25m Normal Started pod/odh-model-controller-769875db56-jr85b Container started 25m Normal Created pod/odh-model-controller-769875db56-jr85b Container created 25m Normal Created pod/maas-controller-5bf8c4d6cc-v9np2 Container created 25m Normal Started pod/kserve-controller-manager-776cc66d47-mvf9p Container started 25m Normal Started pod/maas-controller-5bf8c4d6cc-v9np2 Container started 25m Normal Created pod/kserve-controller-manager-776cc66d47-mvf9p Container created 25m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-jr85b_0dbd78a0-f8a4-4981-a9ce-f0c865af93a0 became leader 25m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5bf8c4d6cc-v9np2_688f4e09-0c93-4afd-9bbf-6c68839edf7c became leader 25m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Container started 25m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Container created 25m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-mvf9p_2b396e79-231d-4fd8-ae04-0226a8645bf2 became leader 25m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw Container image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" already present on machine and can be accessed by the pod 17m Normal Killing pod/maas-controller-5bf8c4d6cc-v9np2 Stopping container manager 17m Normal SuccessfulDelete replicaset/maas-controller-5bf8c4d6cc Deleted pod: maas-controller-5bf8c4d6cc-v9np2 17m Normal SuccessfulCreate replicaset/maas-controller-5bf8c4d6cc Created pod: maas-controller-5bf8c4d6cc-brrh6 17m Normal Pulling pod/maas-controller-5bf8c4d6cc-brrh6 Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 17m Normal AddedInterface pod/maas-controller-5bf8c4d6cc-brrh6 Add eth0 [10.133.0.182/23] from ovn-kubernetes 17m Normal Created pod/maas-controller-5bf8c4d6cc-brrh6 Container created 17m Normal Started pod/maas-controller-5bf8c4d6cc-brrh6 Container started 17m Normal Pulled pod/maas-controller-5bf8c4d6cc-brrh6 Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 3.943s (3.943s including waiting). Image size: 200103442 bytes. 17m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5bf8c4d6cc-brrh6_fc56b497-bcbb-4e90-b2c4-7c12d8a78870 became leader 17m Normal Killing pod/maas-controller-5bf8c4d6cc-brrh6 Stopping container manager 17m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-5bf8c4d6cc from 1 to 0 17m Normal SuccessfulDelete replicaset/maas-controller-5bf8c4d6cc Deleted pod: maas-controller-5bf8c4d6cc-brrh6 17m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-5bf8c4d6cc from 0 to 1 17m Normal SuccessfulCreate replicaset/maas-controller-5bf8c4d6cc Created pod: maas-controller-5bf8c4d6cc-vmh4f 17m Normal Started pod/maas-controller-5bf8c4d6cc-vmh4f Container started 17m Normal Created pod/maas-controller-5bf8c4d6cc-vmh4f Container created 17m Normal Pulled pod/maas-controller-5bf8c4d6cc-vmh4f Successfully pulled image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" in 457ms (457ms including waiting). Image size: 200103442 bytes. 17m Normal Pulling pod/maas-controller-5bf8c4d6cc-vmh4f Pulling image "quay.io/opendatahub/maas-controller:08858542cd3bba18175e02ba9b8418547f22ade2" 17m Normal AddedInterface pod/maas-controller-5bf8c4d6cc-vmh4f Add eth0 [10.133.0.185/23] from ovn-kubernetes 16m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-5bf8c4d6cc-vmh4f_0448db3b-de39-4c15-aea4-fd00b166eb74 became leader