=== events in opendatahub (2026-10-30T18:53:00Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 45m Normal Scheduled pod/llmisvc-controller-manager-6466c554d8-rsmrg Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg to ip-10-0-143-94.ec2.internal 46m Normal Scheduled pod/model-serving-api-75bd89fdfc-49l6m Successfully assigned opendatahub/model-serving-api-75bd89fdfc-49l6m to ip-10-0-133-47.ec2.internal 46m Normal Scheduled pod/llmisvc-controller-manager-869d5dcc5d-g46qt Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt to ip-10-0-143-94.ec2.internal 30m Normal Scheduled pod/test-curl-7265-3647a9 Successfully assigned opendatahub/test-curl-7265-3647a9 to ip-10-0-143-94.ec2.internal 30m Normal Scheduled pod/test-curl-7265-a3d94b Successfully assigned opendatahub/test-curl-7265-a3d94b to ip-10-0-143-94.ec2.internal 43m Normal Scheduled pod/ai-gateway-controller-5bcd7466d5-kzdwm Successfully assigned opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/test-curl-7265-cb4315 Successfully assigned opendatahub/test-curl-7265-cb4315 to ip-10-0-143-94.ec2.internal 45m Normal Scheduled pod/maas-controller-68c97f9884-dsjzk Successfully assigned opendatahub/maas-controller-68c97f9884-dsjzk to ip-10-0-141-106.ec2.internal 45m Normal Scheduled pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s to ip-10-0-143-94.ec2.internal 17m Normal Scheduled pod/maas-controller-68c97f9884-jbsf4 Successfully assigned opendatahub/maas-controller-68c97f9884-jbsf4 to ip-10-0-141-106.ec2.internal 18m Normal Scheduled pod/maas-controller-68c97f9884-qdl7l Successfully assigned opendatahub/maas-controller-68c97f9884-qdl7l to ip-10-0-141-106.ec2.internal 43m Normal Scheduled pod/ai-gateway-controller-b9778b868-g84vm Successfully assigned opendatahub/ai-gateway-controller-b9778b868-g84vm to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/test-curl-7277-8df2b9 Successfully assigned opendatahub/test-curl-7277-8df2b9 to ip-10-0-143-94.ec2.internal 46m Normal Scheduled pod/odh-model-controller-769875db56-84jnk Successfully assigned opendatahub/odh-model-controller-769875db56-84jnk to ip-10-0-133-47.ec2.internal 45m Normal Scheduled pod/maas-controller-579cf7c6d-hp2hf Successfully assigned opendatahub/maas-controller-579cf7c6d-hp2hf to ip-10-0-133-47.ec2.internal 46m Normal Scheduled pod/kserve-controller-manager-776cc66d47-db7c8 Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-db7c8 to ip-10-0-143-94.ec2.internal 46m Normal Scheduled pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9 to ip-10-0-143-94.ec2.internal 26m Normal Scheduled pod/test-curl-7274-90e33e Successfully assigned opendatahub/test-curl-7274-90e33e to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/test-curl-7274-88f432 Successfully assigned opendatahub/test-curl-7274-88f432 to ip-10-0-143-94.ec2.internal 35m Normal Scheduled pod/test-curl-7277-7649da Successfully assigned opendatahub/test-curl-7277-7649da to ip-10-0-143-94.ec2.internal 30m Normal Scheduled pod/test-curl-7265-dfa54a Successfully assigned opendatahub/test-curl-7265-dfa54a to ip-10-0-143-94.ec2.internal 46m Normal ScalingReplicaSet deployment/kserve-module-controller-manager Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1 46m Normal SuccessfulCreate replicaset/kserve-module-controller-manager-7566cf5fc8 Created pod: kserve-module-controller-manager-7566cf5fc8-7prn9 46m Normal Pulling pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" 46m Normal AddedInterface pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Add eth0 [10.132.0.44/23] from ovn-kubernetes 46m Normal Created pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Container created 46m Normal Started pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Container started 46m Normal Pulled pod/kserve-module-controller-manager-7566cf5fc8-7prn9 Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.682s (1.682s including waiting). Image size: 195729000 bytes. 46m Normal LeaderElection lease/kserve-module-controller-manager kserve-module-controller-manager-7566cf5fc8-7prn9_99a8cadb-ef45-4e3c-aa77-980ddb8e331b became leader 46m Normal ScalingReplicaSet deployment/model-serving-api Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1 46m Normal SuccessfulCreate replicaset/model-serving-api-75bd89fdfc Created pod: model-serving-api-75bd89fdfc-49l6m 46m Normal SuccessfulCreate replicaset/kserve-controller-manager-776cc66d47 Created pod: kserve-controller-manager-776cc66d47-db7c8 46m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-869d5dcc5d Created pod: llmisvc-controller-manager-869d5dcc5d-g46qt 46m Normal ScalingReplicaSet deployment/kserve-controller-manager Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1 46m Normal ScalingReplicaSet deployment/odh-model-controller Scaled up replica set odh-model-controller-769875db56 from 0 to 1 46m Normal SuccessfulCreate replicaset/odh-model-controller-769875db56 Created pod: odh-model-controller-769875db56-84jnk 46m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1 46m Normal AddedInterface pod/kserve-controller-manager-776cc66d47-db7c8 Add eth0 [10.132.0.46/23] from ovn-kubernetes 46m Normal Pulling pod/kserve-controller-manager-776cc66d47-db7c8 Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" 46m Normal Pulling pod/model-serving-api-75bd89fdfc-49l6m Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" 46m Normal AddedInterface pod/llmisvc-controller-manager-869d5dcc5d-g46qt Add eth0 [10.132.0.47/23] from ovn-kubernetes 46m Normal AddedInterface pod/model-serving-api-75bd89fdfc-49l6m Add eth0 [10.134.0.37/23] from ovn-kubernetes 46m Normal Pulling pod/llmisvc-controller-manager-869d5dcc5d-g46qt Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 46m Normal Pulling pod/odh-model-controller-769875db56-84jnk Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" 46m Normal AddedInterface pod/odh-model-controller-769875db56-84jnk Add eth0 [10.134.0.38/23] from ovn-kubernetes 45m Normal Started pod/model-serving-api-75bd89fdfc-49l6m Container started 45m Normal Started pod/kserve-controller-manager-776cc66d47-db7c8 Container started 45m Normal Created pod/kserve-controller-manager-776cc66d47-db7c8 Container created 45m Normal Created pod/model-serving-api-75bd89fdfc-49l6m Container created 45m Normal Started pod/llmisvc-controller-manager-869d5dcc5d-g46qt Container started 45m Normal Created pod/llmisvc-controller-manager-869d5dcc5d-g46qt Container created 45m Normal Pulled pod/llmisvc-controller-manager-869d5dcc5d-g46qt Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.196s (2.196s including waiting). Image size: 235100981 bytes. 45m Normal Pulled pod/kserve-controller-manager-776cc66d47-db7c8 Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.395s (2.395s including waiting). Image size: 247676581 bytes. 45m Normal Pulled pod/model-serving-api-75bd89fdfc-49l6m Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.2s (2.2s including waiting). Image size: 184925761 bytes. 45m Normal Pulled pod/odh-model-controller-769875db56-84jnk Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.239s (3.239s including waiting). Image size: 236755567 bytes. 45m Normal Started pod/odh-model-controller-769875db56-84jnk Container started 45m Normal Created pod/odh-model-controller-769875db56-84jnk Container created 45m Normal LeaderElection lease/odh-model-controller.opendatahub.io odh-model-controller-769875db56-84jnk_141a299a-58d5-4d32-91e6-4d2ca82eba28 became leader 45m Normal LeaderElection lease/kserve-controller-manager-leader-lock kserve-controller-manager-776cc66d47-db7c8_2644c615-af82-4535-a8fe-476df29bc4ac became leader 45m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-869d5dcc5d-g46qt_00021e76-d124-4117-b134-60d77afb7a33 became leader 45m Normal Pulling pod/maas-controller-579cf7c6d-hp2hf Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 45m Normal SuccessfulCreate replicaset/maas-controller-579cf7c6d Created pod: maas-controller-579cf7c6d-hp2hf 45m Normal SuccessfulCreate replicaset/maas-controller-68c97f9884 Created pod: maas-controller-68c97f9884-dsjzk 45m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-579cf7c6d from 0 to 1 45m Normal AddedInterface pod/maas-controller-579cf7c6d-hp2hf Add eth0 [10.134.0.39/23] from ovn-kubernetes 45m Normal AddedInterface pod/maas-controller-68c97f9884-dsjzk Add eth0 [10.133.0.21/23] from ovn-kubernetes 45m Normal Pulled pod/maas-controller-68c97f9884-dsjzk Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.29s (2.29s including waiting). Image size: 200053271 bytes. 45m Normal Created pod/maas-controller-579cf7c6d-hp2hf Container created 45m Normal Started pod/maas-controller-579cf7c6d-hp2hf Container started 45m Normal Pulled pod/maas-controller-579cf7c6d-hp2hf Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 2.836s (2.836s including waiting). Image size: 200053271 bytes. 45m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-68c97f9884-dsjzk_83f80fef-1bfa-4ae9-aaab-e41a264a88b5 became leader 45m Warning FailedMount pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found 45m Normal SuccessfulCreate replicaset/usage-logs-tenancy-proxy-9c5d49bcd Created pod: usage-logs-tenancy-proxy-9c5d49bcd-vgw6s 45m Normal ScalingReplicaSet deployment/usage-logs-tenancy-proxy Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1 45m Normal AddedInterface pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Add eth0 [10.132.0.48/23] from ovn-kubernetes 45m Normal Pulling pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" 45m Normal Killing pod/maas-controller-579cf7c6d-hp2hf Stopping container manager 45m Normal SuccessfulDelete replicaset/maas-controller-579cf7c6d Deleted pod: maas-controller-579cf7c6d-hp2hf 45m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-579cf7c6d from 1 to 0 45m Normal Pulled pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 10.779s (10.779s including waiting). Image size: 1089502687 bytes. 45m Normal Created pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Container created 45m Normal Started pod/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s Container started 45m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1 45m Normal SuccessfulCreate replicaset/llmisvc-controller-manager-6466c554d8 Created pod: llmisvc-controller-manager-6466c554d8-rsmrg 45m Normal Pulled pod/llmisvc-controller-manager-6466c554d8-rsmrg Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 346ms (346ms including waiting). Image size: 235100981 bytes. 45m Normal Created pod/llmisvc-controller-manager-6466c554d8-rsmrg Container created 45m Normal Pulling pod/llmisvc-controller-manager-6466c554d8-rsmrg Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" 45m Normal AddedInterface pod/llmisvc-controller-manager-6466c554d8-rsmrg Add eth0 [10.132.0.51/23] from ovn-kubernetes 45m Normal Started pod/llmisvc-controller-manager-6466c554d8-rsmrg Container started 44m Normal ScalingReplicaSet deployment/llmisvc-controller-manager Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0 44m Normal Killing pod/llmisvc-controller-manager-869d5dcc5d-g46qt Stopping container manager 44m Normal SuccessfulDelete replicaset/llmisvc-controller-manager-869d5dcc5d Deleted pod: llmisvc-controller-manager-869d5dcc5d-g46qt 44m Normal LeaderElection lease/llminferenceservice-kserve-controller-manager llmisvc-controller-manager-6466c554d8-rsmrg_ac49c5b1-d84c-495b-a667-a02fd90a1c84 became leader 43m Normal SuccessfulCreate replicaset/ai-gateway-controller-b9778b868 Created pod: ai-gateway-controller-b9778b868-g84vm 43m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-b9778b868 from 0 to 1 43m Normal Pulling pod/ai-gateway-controller-b9778b868-g84vm Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-200" 43m Normal AddedInterface pod/ai-gateway-controller-b9778b868-g84vm Add eth0 [10.133.0.28/23] from ovn-kubernetes 43m Normal Started pod/ai-gateway-controller-b9778b868-g84vm Container started 43m Normal Created pod/ai-gateway-controller-b9778b868-g84vm Container created 43m Normal Pulled pod/ai-gateway-controller-b9778b868-g84vm Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-200" in 1.676s (1.676s including waiting). Image size: 183888316 bytes. 43m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-b9778b868-g84vm_53592642-8167-4a35-a003-6bb8b3196cab became leader 43m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled up replica set ai-gateway-controller-5bcd7466d5 from 0 to 1 43m Normal SuccessfulCreate replicaset/ai-gateway-controller-5bcd7466d5 Created pod: ai-gateway-controller-5bcd7466d5-kzdwm 43m Normal AddedInterface pod/ai-gateway-controller-5bcd7466d5-kzdwm Add eth0 [10.134.0.55/23] from ovn-kubernetes 43m Normal Pulling pod/ai-gateway-controller-5bcd7466d5-kzdwm Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-200" 43m Normal Started pod/ai-gateway-controller-5bcd7466d5-kzdwm Container started 43m Normal Pulled pod/ai-gateway-controller-5bcd7466d5-kzdwm Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-200" in 1.459s (1.459s including waiting). Image size: 183888316 bytes. 43m Normal Created pod/ai-gateway-controller-5bcd7466d5-kzdwm Container created 43m Normal SuccessfulDelete replicaset/ai-gateway-controller-b9778b868 Deleted pod: ai-gateway-controller-b9778b868-g84vm 43m Normal Killing pod/ai-gateway-controller-b9778b868-g84vm Stopping container manager 43m Normal ScalingReplicaSet deployment/ai-gateway-controller Scaled down replica set ai-gateway-controller-b9778b868 from 1 to 0 42m Normal LeaderElection lease/ai-gateway-controller-leader.opendatahub.io ai-gateway-controller-5bcd7466d5-kzdwm_58c45e9a-15d0-41ba-b03f-8d1f82652be4 became leader 39m Warning Unhealthy pod/model-serving-api-75bd89fdfc-49l6m Liveness probe failed: Get "https://10.134.0.37:8443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 35m Normal AddedInterface pod/test-curl-7277-8df2b9 Add eth0 [10.132.0.90/23] from ovn-kubernetes 35m Normal Pulling pod/test-curl-7277-8df2b9 Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 35m Normal Started pod/test-curl-7277-8df2b9 Container started 35m Normal Created pod/test-curl-7277-8df2b9 Container created 35m Normal Pulled pod/test-curl-7277-8df2b9 Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 1.997s (1.997s including waiting). Image size: 106100986 bytes. 35m Normal Killing pod/test-curl-7277-8df2b9 Stopping container test-curl-7277-8df2b9 35m Normal AddedInterface pod/test-curl-7277-7649da Add eth0 [10.132.0.91/23] from ovn-kubernetes 35m Normal Pulled pod/test-curl-7277-7649da Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 35m Normal Created pod/test-curl-7277-7649da Container created 35m Normal Started pod/test-curl-7277-7649da Container started 35m Normal Killing pod/test-curl-7277-7649da Stopping container test-curl-7277-7649da 30m Normal AddedInterface pod/test-curl-7265-dfa54a Add eth0 [10.132.0.107/23] from ovn-kubernetes 30m Normal Started pod/test-curl-7265-dfa54a Container started 30m Normal Pulled pod/test-curl-7265-dfa54a Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal Created pod/test-curl-7265-dfa54a Container created 30m Normal Started pod/test-curl-7265-cb4315 Container started 30m Normal Created pod/test-curl-7265-cb4315 Container created 30m Normal Pulled pod/test-curl-7265-cb4315 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/test-curl-7265-cb4315 Add eth0 [10.132.0.108/23] from ovn-kubernetes 30m Normal Killing pod/test-curl-7265-dfa54a Stopping container test-curl-7265-dfa54a 30m Normal Started pod/test-curl-7265-a3d94b Container started 30m Normal Killing pod/test-curl-7265-cb4315 Stopping container test-curl-7265-cb4315 30m Normal AddedInterface pod/test-curl-7265-a3d94b Add eth0 [10.132.0.109/23] from ovn-kubernetes 30m Normal Pulled pod/test-curl-7265-a3d94b Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal Created pod/test-curl-7265-a3d94b Container created 30m Normal Killing pod/test-curl-7265-a3d94b Stopping container test-curl-7265-a3d94b 30m Normal AddedInterface pod/test-curl-7265-3647a9 Add eth0 [10.132.0.110/23] from ovn-kubernetes 30m Normal Pulled pod/test-curl-7265-3647a9 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 30m Normal Created pod/test-curl-7265-3647a9 Container created 30m Normal Started pod/test-curl-7265-3647a9 Container started 30m Normal Killing pod/test-curl-7265-3647a9 Stopping container test-curl-7265-3647a9 30m Warning NodeNotReady pod/model-serving-api-75bd89fdfc-49l6m Node is not ready 30m Warning NodeNotReady pod/odh-model-controller-769875db56-84jnk Node is not ready 30m Warning NodeNotReady pod/ai-gateway-controller-5bcd7466d5-kzdwm Node is not ready 30m Normal TaintManagerEviction pod/ai-gateway-controller-5bcd7466d5-kzdwm Cancelling deletion of Pod opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm 30m Normal TaintManagerEviction pod/model-serving-api-75bd89fdfc-49l6m Cancelling deletion of Pod opendatahub/model-serving-api-75bd89fdfc-49l6m 30m Normal TaintManagerEviction pod/odh-model-controller-769875db56-84jnk Cancelling deletion of Pod opendatahub/odh-model-controller-769875db56-84jnk 30m Normal Pulling pod/maas-controller-68c97f9884-dsjzk Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 30m Normal Created pod/maas-controller-68c97f9884-dsjzk Container created 30m Normal Started pod/maas-controller-68c97f9884-dsjzk Container started 30m Normal Pulled pod/maas-controller-68c97f9884-dsjzk Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 391ms (391ms including waiting). Image size: 200053271 bytes. 29m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-68c97f9884-dsjzk_c5b3fbcc-b371-4c3a-8384-55b2e34ae1c9 became leader 29m Normal AddedInterface pod/test-curl-7274-88f432 Add eth0 [10.132.0.116/23] from ovn-kubernetes 29m Normal Pulled pod/test-curl-7274-88f432 Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod 29m Normal Started pod/test-curl-7274-88f432 Container started 29m Normal Created pod/test-curl-7274-88f432 Container created 29m Normal Killing pod/test-curl-7274-88f432 Stopping container test-curl-7274-88f432 28m Warning Unhealthy pod/odh-model-controller-769875db56-84jnk Liveness probe failed: Get "http://10.134.0.38:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Warning Unhealthy pod/ai-gateway-controller-5bcd7466d5-kzdwm Liveness probe failed: Get "http://10.134.0.55:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Warning Unhealthy pod/model-serving-api-75bd89fdfc-49l6m Readiness probe failed: Get "https://10.134.0.37:8443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 28m Warning Unhealthy pod/model-serving-api-75bd89fdfc-49l6m Liveness probe failed: Get "https://10.134.0.37:8443/healthz": context deadline exceeded 28m Warning Unhealthy pod/ai-gateway-controller-5bcd7466d5-kzdwm Readiness probe failed: Get "http://10.134.0.55:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Warning Unhealthy pod/model-serving-api-75bd89fdfc-49l6m Readiness probe failed: Get "https://10.134.0.37:8443/readyz": context deadline exceeded 28m Warning Unhealthy pod/odh-model-controller-769875db56-84jnk Readiness probe failed: Get "http://10.134.0.38:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal Pulling pod/test-curl-7274-90e33e Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" 26m Normal AddedInterface pod/test-curl-7274-90e33e Add eth0 [10.133.0.194/23] from ovn-kubernetes 26m Normal Created pod/test-curl-7274-90e33e Container created 26m Normal Started pod/test-curl-7274-90e33e Container started 26m Normal Pulled pod/test-curl-7274-90e33e Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 4.336s (4.336s including waiting). Image size: 106100986 bytes. 26m Normal Killing pod/test-curl-7274-90e33e Stopping container test-curl-7274-90e33e 18m Normal SuccessfulDelete replicaset/maas-controller-68c97f9884 Deleted pod: maas-controller-68c97f9884-dsjzk 18m Normal Killing pod/maas-controller-68c97f9884-dsjzk Stopping container manager 18m Normal SuccessfulCreate replicaset/maas-controller-68c97f9884 Created pod: maas-controller-68c97f9884-qdl7l 18m Normal Pulled pod/maas-controller-68c97f9884-qdl7l Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 491ms (491ms including waiting). Image size: 200053271 bytes. 18m Normal Pulling pod/maas-controller-68c97f9884-qdl7l Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 18m Normal AddedInterface pod/maas-controller-68c97f9884-qdl7l Add eth0 [10.133.0.245/23] from ovn-kubernetes 18m Normal Started pod/maas-controller-68c97f9884-qdl7l Container started 18m Normal Created pod/maas-controller-68c97f9884-qdl7l Container created 18m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-68c97f9884-qdl7l_8e08076b-2e2c-44a7-9885-6cf39fc9057d became leader 17m Normal ScalingReplicaSet deployment/maas-controller Scaled down replica set maas-controller-68c97f9884 from 1 to 0 17m Normal SuccessfulDelete replicaset/maas-controller-68c97f9884 Deleted pod: maas-controller-68c97f9884-qdl7l 17m Normal Killing pod/maas-controller-68c97f9884-qdl7l Stopping container manager 17m Normal ScalingReplicaSet deployment/maas-controller Scaled up replica set maas-controller-68c97f9884 from 0 to 1 17m Normal SuccessfulCreate replicaset/maas-controller-68c97f9884 Created pod: maas-controller-68c97f9884-jbsf4 17m Normal AddedInterface pod/maas-controller-68c97f9884-jbsf4 Add eth0 [10.133.0.249/23] from ovn-kubernetes 17m Normal Started pod/maas-controller-68c97f9884-jbsf4 Container started 17m Normal Created pod/maas-controller-68c97f9884-jbsf4 Container created 17m Normal Pulling pod/maas-controller-68c97f9884-jbsf4 Pulling image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" 17m Normal Pulled pod/maas-controller-68c97f9884-jbsf4 Successfully pulled image "quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56" in 359ms (359ms including waiting). Image size: 200053271 bytes. 17m Normal LeaderElection lease/maas-controller.models-as-a-service.opendatahub.io maas-controller-68c97f9884-jbsf4_d40e9252-aa1a-4c5b-a202-ff938992c43d became leader