=== events in odh-ai-gateway-infra (2026-11-04T05:47:43Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 3m10s Normal Scheduled pod/maas-api-revoke-sub-e3392fc5-6r58z Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-e3392fc5-6r58z to ip-10-0-141-135.us-east-2.compute.internal 2m53s Normal Scheduled pod/maas-api-revoke-sub-b83493ec-k54hm Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-b83493ec-k54hm to ip-10-0-133-170.us-east-2.compute.internal 5m14s Normal Scheduled pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 to ip-10-0-128-126.us-east-2.compute.internal 3m7s Normal Scheduled pod/maas-api-revoke-sub-71199809-hjkw7 Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-71199809-hjkw7 to ip-10-0-141-135.us-east-2.compute.internal 2m28s Normal Scheduled pod/maas-api-revoke-sub-3d8b0bd4-r84p7 Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-3d8b0bd4-r84p7 to ip-10-0-141-135.us-east-2.compute.internal 10m Normal Scheduled pod/postgres-7cdbf6dc44-2bpbj Successfully assigned odh-ai-gateway-infra/postgres-7cdbf6dc44-2bpbj to ip-10-0-128-126.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-29896185-6smnz Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-29896185-6smnz to ip-10-0-133-170.us-east-2.compute.internal 9m10s Normal Scheduled pod/maas-api-674d648f7b-7s9tq Successfully assigned odh-ai-gateway-infra/maas-api-674d648f7b-7s9tq to ip-10-0-128-126.us-east-2.compute.internal 5m13s Normal Scheduled pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv to ip-10-0-128-126.us-east-2.compute.internal 3m16s Normal Scheduled pod/maas-api-revoke-sub-3dff2ea7-c6xcz Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-3dff2ea7-c6xcz to ip-10-0-141-135.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg to ip-10-0-133-170.us-east-2.compute.internal 5m16s Normal Scheduled pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l to ip-10-0-128-126.us-east-2.compute.internal 5m9s Normal Scheduled pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s to ip-10-0-128-126.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q to ip-10-0-141-135.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd to ip-10-0-141-135.us-east-2.compute.internal 2m31s Normal Scheduled pod/maas-api-revoke-sub-40e8492f-z5ngh Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-40e8492f-z5ngh to ip-10-0-133-170.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq to ip-10-0-141-135.us-east-2.compute.internal 5m12s Normal Scheduled pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m to ip-10-0-128-126.us-east-2.compute.internal 5m12s Normal Scheduled pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm to ip-10-0-133-170.us-east-2.compute.internal 2m16s Normal Scheduled pod/maas-api-revoke-sub-e76d4112-zgmv6 Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-e76d4112-zgmv6 to ip-10-0-141-135.us-east-2.compute.internal 2m54s Normal Scheduled pod/maas-api-revoke-sub-a24ab2f5-l6mwl Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-a24ab2f5-l6mwl to ip-10-0-141-135.us-east-2.compute.internal 9m Normal Scheduled pod/maas-api-5d8dd6bcb9-tbdvp Successfully assigned odh-ai-gateway-infra/maas-api-5d8dd6bcb9-tbdvp to ip-10-0-128-126.us-east-2.compute.internal 2m15s Normal Scheduled pod/maas-api-revoke-sub-93ee4734-c57px Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-93ee4734-c57px to ip-10-0-133-170.us-east-2.compute.internal 3m1s Normal Scheduled pod/maas-api-revoke-sub-94658290-8vppk Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-94658290-8vppk to ip-10-0-141-135.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx to ip-10-0-133-170.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl to ip-10-0-141-135.us-east-2.compute.internal 2m32s Normal Scheduled pod/maas-api-revoke-sub-ab03cae5-6dmpb Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-ab03cae5-6dmpb to ip-10-0-141-135.us-east-2.compute.internal 5m10s Normal Scheduled pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Successfully assigned odh-ai-gateway-infra/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 to ip-10-0-141-135.us-east-2.compute.internal 2m14s Normal Scheduled pod/maas-api-revoke-sub-6aeadf02-q2dnz Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-6aeadf02-q2dnz to ip-10-0-141-135.us-east-2.compute.internal 2m41s Normal Scheduled pod/maas-api-revoke-sub-cb56d506-fl25h Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-cb56d506-fl25h to ip-10-0-141-135.us-east-2.compute.internal 2m32s Normal Scheduled pod/maas-api-revoke-sub-25dd93bf-8zbsp Successfully assigned odh-ai-gateway-infra/maas-api-revoke-sub-25dd93bf-8zbsp to ip-10-0-141-135.us-east-2.compute.internal 2m43s Normal Scheduled pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Successfully assigned odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 to ip-10-0-133-170.us-east-2.compute.internal 10m Normal ScalingReplicaSet deployment/postgres Scaled up replica set postgres-7cdbf6dc44 from 0 to 1 10m Normal SuccessfulCreate replicaset/postgres-7cdbf6dc44 Created pod: postgres-7cdbf6dc44-2bpbj 10m Normal AddedInterface pod/postgres-7cdbf6dc44-2bpbj Add eth0 [10.134.0.33/23] from ovn-kubernetes 10m Normal Pulling pod/postgres-7cdbf6dc44-2bpbj Pulling image "registry.redhat.io/rhel9/postgresql-16:latest" 9m57s Normal Started pod/postgres-7cdbf6dc44-2bpbj Container started 9m57s Normal Created pod/postgres-7cdbf6dc44-2bpbj Container created 9m57s Normal Pulled pod/postgres-7cdbf6dc44-2bpbj Successfully pulled image "registry.redhat.io/rhel9/postgresql-16:latest" in 13.331s (13.331s including waiting). Image size: 403732708 bytes. 9m11s Normal ScalingReplicaSet deployment/maas-api Scaled up replica set maas-api-674d648f7b from 0 to 1 9m10s Normal SuccessfulCreate replicaset/maas-api-674d648f7b Created pod: maas-api-674d648f7b-7s9tq 9m10s Warning FailedCreate replicaset/maas-api-674d648f7b Error creating: pods "maas-api-674d648f7b-" is forbidden: error looking up service account odh-ai-gateway-infra/maas-api: serviceaccount "maas-api" not found 9m10s Normal AddedInterface pod/maas-api-674d648f7b-7s9tq Add eth0 [10.134.0.38/23] from ovn-kubernetes 9m10s Normal Pulling pod/maas-api-674d648f7b-7s9tq Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 9m8s Normal Pulled pod/maas-api-674d648f7b-7s9tq Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 1.525s (1.525s including waiting). Image size: 194589557 bytes. 9m8s Normal Started pod/maas-api-674d648f7b-7s9tq Container started 9m8s Normal Created pod/maas-api-674d648f7b-7s9tq Container created 9m1s Normal ScalingReplicaSet deployment/maas-api Scaled up replica set maas-api-5d8dd6bcb9 from 0 to 1 9m1s Normal SuccessfulCreate replicaset/maas-api-5d8dd6bcb9 Created pod: maas-api-5d8dd6bcb9-tbdvp 9m Normal AddedInterface pod/maas-api-5d8dd6bcb9-tbdvp Add eth0 [10.134.0.39/23] from ovn-kubernetes 9m Normal Pulling pod/maas-api-5d8dd6bcb9-tbdvp Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 9m Normal Pulled pod/maas-api-5d8dd6bcb9-tbdvp Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 404ms (404ms including waiting). Image size: 194589557 bytes. 9m Normal Created pod/maas-api-5d8dd6bcb9-tbdvp Container created 9m Normal Started pod/maas-api-5d8dd6bcb9-tbdvp Container started 8m53s Normal SuccessfulDelete replicaset/maas-api-674d648f7b Deleted pod: maas-api-674d648f7b-7s9tq 8m53s Normal Killing pod/maas-api-674d648f7b-7s9tq Stopping container maas-api 8m53s Normal ScalingReplicaSet deployment/maas-api Scaled down replica set maas-api-674d648f7b from 1 to 0 5m17s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w3-f910f5-86f6b57569 Created pod: maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l 5m17s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w3-f910f5 Scaled up replica set maas-api-e2e-worker-w3-f910f5-86f6b57569 from 0 to 1 5m16s Normal AddedInterface pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Add eth0 [10.134.0.44/23] from ovn-kubernetes 5m16s Normal Pulled pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 534ms (534ms including waiting). Image size: 194589557 bytes. 5m16s Normal Pulling pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m15s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w6-2b14d0-6749f7ff76 Created pod: maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 5m15s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w6-2b14d0 Scaled up replica set maas-api-e2e-worker-w6-2b14d0-6749f7ff76 from 0 to 1 5m15s Normal Started pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Container started 5m15s Normal Created pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Container created 5m14s Normal Created pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Container created 5m14s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w4-9a0c8b Scaled up replica set maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b from 0 to 1 5m14s Normal Started pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Container started 5m14s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b Created pod: maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv 5m14s Normal Pulled pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 454ms (454ms including waiting). Image size: 194589557 bytes. 5m14s Normal Pulling pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m14s Normal AddedInterface pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Add eth0 [10.134.0.46/23] from ovn-kubernetes 5m13s Normal Pulled pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 428ms (428ms including waiting). Image size: 194589557 bytes. 5m13s Normal Pulling pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m13s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w0-27935d Scaled up replica set maas-api-e2e-worker-w0-27935d-54dd4977df from 0 to 1 5m13s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w0-27935d-54dd4977df Created pod: maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m 5m13s Normal AddedInterface pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Add eth0 [10.134.0.47/23] from ovn-kubernetes 5m13s Normal Created pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Container created 5m13s Warning FailedMount pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-worker-w0-27935d-serving-cert" not found 5m13s Normal Started pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Container started 5m12s Normal AddedInterface pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Add eth0 [10.134.0.48/23] from ovn-kubernetes 5m12s Normal Pulling pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m12s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w5-9d808e Scaled up replica set maas-api-e2e-worker-w5-9d808e-7788c7bbdd from 0 to 1 5m12s Normal Pulling pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m12s Normal AddedInterface pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Add eth0 [10.133.0.43/23] from ovn-kubernetes 5m12s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w5-9d808e-7788c7bbdd Created pod: maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm 5m11s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c Created pod: maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 5m11s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w1-f9b4a9 Scaled up replica set maas-api-e2e-worker-w1-f9b4a9-97bfbf58c from 0 to 1 5m11s Normal Pulled pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 364ms (364ms including waiting). Image size: 194589557 bytes. 5m11s Normal Created pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Container created 5m11s Normal Started pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Container started 5m10s Warning FailedMount pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-worker-w2-ade2e8-serving-cert" not found 5m10s Normal ScalingReplicaSet deployment/maas-api-e2e-worker-w2-ade2e8 Scaled up replica set maas-api-e2e-worker-w2-ade2e8-6b8b9798bb from 0 to 1 5m10s Normal SuccessfulCreate replicaset/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb Created pod: maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s 5m10s Normal AddedInterface pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Add eth0 [10.132.0.68/23] from ovn-kubernetes 5m10s Normal Pulling pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m9s Normal AddedInterface pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Add eth0 [10.134.0.49/23] from ovn-kubernetes 5m9s Normal Pulling pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Pulling image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" 5m9s Normal Started pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Container started 5m9s Normal Created pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Container created 5m9s Normal Pulled pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 2.769s (2.769s including waiting). Image size: 194589557 bytes. 5m8s Normal Started pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Container started 5m8s Normal Created pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Container created 5m8s Normal Pulled pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 403ms (403ms including waiting). Image size: 194589557 bytes. 5m7s Normal Pulled pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Successfully pulled image "quay.io/opendatahub/maas-api:08858542cd3bba18175e02ba9b8418547f22ade2" in 3.597s (3.597s including waiting). Image size: 194589557 bytes. 5m6s Normal Created pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Container created 5m6s Normal Started pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Container started 3m17s Normal SuccessfulCreate job/maas-api-revoke-sub-3dff2ea7 Created pod: maas-api-revoke-sub-3dff2ea7-c6xcz 3m16s Normal AddedInterface pod/maas-api-revoke-sub-3dff2ea7-c6xcz Add eth0 [10.132.0.77/23] from ovn-kubernetes 3m16s Normal Pulling pod/maas-api-revoke-sub-3dff2ea7-c6xcz Pulling image "registry.redhat.io/ubi9/ubi-minimal:9.7" 3m13s Normal Started pod/maas-api-revoke-sub-3dff2ea7-c6xcz Container started 3m13s Normal Created pod/maas-api-revoke-sub-3dff2ea7-c6xcz Container created 3m13s Normal Pulled pod/maas-api-revoke-sub-3dff2ea7-c6xcz Successfully pulled image "registry.redhat.io/ubi9/ubi-minimal:9.7" in 3.051s (3.051s including waiting). Image size: 107195206 bytes. 3m11s Normal SuccessfulCreate job/maas-api-revoke-sub-e3392fc5 Created pod: maas-api-revoke-sub-e3392fc5-6r58z 3m10s Normal Created pod/maas-api-revoke-sub-e3392fc5-6r58z Container created 3m10s Normal Pulled pod/maas-api-revoke-sub-e3392fc5-6r58z Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 3m10s Normal AddedInterface pod/maas-api-revoke-sub-e3392fc5-6r58z Add eth0 [10.132.0.80/23] from ovn-kubernetes 3m10s Normal Started pod/maas-api-revoke-sub-e3392fc5-6r58z Container started 3m10s Normal Completed job/maas-api-revoke-sub-3dff2ea7 Job completed 3m8s Normal SuccessfulCreate job/maas-api-revoke-sub-71199809 Created pod: maas-api-revoke-sub-71199809-hjkw7 3m7s Normal Pulled pod/maas-api-revoke-sub-71199809-hjkw7 Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 3m7s Normal Started pod/maas-api-revoke-sub-71199809-hjkw7 Container started 3m7s Normal Created pod/maas-api-revoke-sub-71199809-hjkw7 Container created 3m7s Normal Completed job/maas-api-revoke-sub-e3392fc5 Job completed 3m7s Normal AddedInterface pod/maas-api-revoke-sub-71199809-hjkw7 Add eth0 [10.132.0.81/23] from ovn-kubernetes 3m4s Normal Completed job/maas-api-revoke-sub-71199809 Job completed 3m1s Normal SuccessfulCreate job/maas-api-revoke-sub-94658290 Created pod: maas-api-revoke-sub-94658290-8vppk 3m1s Normal AddedInterface pod/maas-api-revoke-sub-94658290-8vppk Add eth0 [10.132.0.82/23] from ovn-kubernetes 3m1s Normal Pulled pod/maas-api-revoke-sub-94658290-8vppk Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 3m Normal Created pod/maas-api-revoke-sub-94658290-8vppk Container created 3m Normal Started pod/maas-api-revoke-sub-94658290-8vppk Container started 2m57s Normal Completed job/maas-api-revoke-sub-94658290 Job completed 2m55s Normal SuccessfulCreate job/maas-api-revoke-sub-a24ab2f5 Created pod: maas-api-revoke-sub-a24ab2f5-l6mwl 2m54s Normal AddedInterface pod/maas-api-revoke-sub-a24ab2f5-l6mwl Add eth0 [10.132.0.83/23] from ovn-kubernetes 2m54s Normal Pulled pod/maas-api-revoke-sub-a24ab2f5-l6mwl Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m54s Normal Created pod/maas-api-revoke-sub-a24ab2f5-l6mwl Container created 2m54s Normal Started pod/maas-api-revoke-sub-a24ab2f5-l6mwl Container started 2m53s Normal Pulling pod/maas-api-revoke-sub-b83493ec-k54hm Pulling image "registry.redhat.io/ubi9/ubi-minimal:9.7" 2m53s Normal SuccessfulCreate job/maas-api-revoke-sub-b83493ec Created pod: maas-api-revoke-sub-b83493ec-k54hm 2m53s Normal AddedInterface pod/maas-api-revoke-sub-b83493ec-k54hm Add eth0 [10.133.0.54/23] from ovn-kubernetes 2m51s Normal Completed job/maas-api-revoke-sub-a24ab2f5 Job completed 2m50s Normal Started pod/maas-api-revoke-sub-b83493ec-k54hm Container started 2m50s Normal Pulled pod/maas-api-revoke-sub-b83493ec-k54hm Successfully pulled image "registry.redhat.io/ubi9/ubi-minimal:9.7" in 2.486s (2.486s including waiting). Image size: 107195206 bytes. 2m50s Normal Created pod/maas-api-revoke-sub-b83493ec-k54hm Container created 2m47s Normal Completed job/maas-api-revoke-sub-b83493ec Job completed 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w6-2b14d0 Created job maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w5-9d808e Created job maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185 2m43s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Add eth0 [10.133.0.58/23] from ovn-kubernetes 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w4-9a0c8b Created job maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx 2m43s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Add eth0 [10.133.0.59/23] from ovn-kubernetes 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-29896185 Created pod: maas-api-key-cleanup-29896185-6smnz 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w2-ade2e8 Created job maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185 2m43s Normal AddedInterface pod/maas-api-key-cleanup-29896185-6smnz Add eth0 [10.133.0.57/23] from ovn-kubernetes 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup Created job maas-api-key-cleanup-29896185 2m43s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Add eth0 [10.133.0.56/23] from ovn-kubernetes 2m43s Normal Killing pod/maas-api-e2e-worker-w2-ade2e8-6b8b9798bb-zxl4s Stopping container maas-api 2m43s Normal Killing pod/maas-api-e2e-worker-w1-f9b4a9-97bfbf58c-rf545 Stopping container maas-api 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w0-27935d Created job maas-api-key-cleanup-e2e-worker-w0-27935d-29896185 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w3-f910f5 Created job maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl 2m43s Normal SuccessfulCreate cronjob/maas-api-key-cleanup-e2e-worker-w1-f9b4a9 Created job maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185 2m43s Normal SuccessfulCreate job/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185 Created pod: maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd 2m42s Normal Created pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Container created 2m42s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Add eth0 [10.132.0.86/23] from ovn-kubernetes 2m42s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Add eth0 [10.132.0.88/23] from ovn-kubernetes 2m42s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd Add eth0 [10.132.0.85/23] from ovn-kubernetes 2m42s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m42s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m42s Normal Started pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Container started 2m42s Normal AddedInterface pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Add eth0 [10.132.0.87/23] from ovn-kubernetes 2m42s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m41s Normal AddedInterface pod/maas-api-revoke-sub-cb56d506-fl25h Add eth0 [10.132.0.89/23] from ovn-kubernetes 2m41s Normal Pulled pod/maas-api-revoke-sub-cb56d506-fl25h Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m41s Normal Created pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Container created 2m41s Normal Started pod/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd Container started 2m41s Normal Created pod/maas-api-key-cleanup-e2e-worker-w2-ade2e8-29896185-b7pmd Container created 2m41s Normal Started pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Container started 2m41s Normal Killing pod/maas-api-e2e-worker-w4-9a0c8b-6bd5b9798b-crjlv Stopping container maas-api 2m41s Normal Killing pod/maas-api-key-cleanup-e2e-worker-w1-f9b4a9-29896185-lggdl Stopping container cleanup 2m41s Normal SuccessfulCreate job/maas-api-revoke-sub-cb56d506 Created pod: maas-api-revoke-sub-cb56d506-fl25h 2m40s Normal Started pod/maas-api-revoke-sub-cb56d506-fl25h Container started 2m40s Normal Killing pod/maas-api-key-cleanup-e2e-worker-w4-9a0c8b-29896185-w8842 Stopping container cleanup 2m40s Normal Created pod/maas-api-revoke-sub-cb56d506-fl25h Container created 2m37s Normal Completed job/maas-api-revoke-sub-cb56d506 Job completed 2m33s Normal SuccessfulCreate job/maas-api-revoke-sub-25dd93bf Created pod: maas-api-revoke-sub-25dd93bf-8zbsp 2m33s Normal SuccessfulCreate job/maas-api-revoke-sub-ab03cae5 Created pod: maas-api-revoke-sub-ab03cae5-6dmpb 2m32s Normal AddedInterface pod/maas-api-revoke-sub-ab03cae5-6dmpb Add eth0 [10.132.0.92/23] from ovn-kubernetes 2m32s Normal Pulled pod/maas-api-revoke-sub-ab03cae5-6dmpb Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m32s Normal AddedInterface pod/maas-api-revoke-sub-25dd93bf-8zbsp Add eth0 [10.132.0.93/23] from ovn-kubernetes 2m31s Normal Created pod/maas-api-revoke-sub-ab03cae5-6dmpb Container created 2m31s Normal Pulled pod/maas-api-revoke-sub-40e8492f-z5ngh Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m31s Normal Started pod/maas-api-revoke-sub-25dd93bf-8zbsp Container started 2m31s Normal AddedInterface pod/maas-api-revoke-sub-40e8492f-z5ngh Add eth0 [10.133.0.61/23] from ovn-kubernetes 2m31s Normal Started pod/maas-api-revoke-sub-ab03cae5-6dmpb Container started 2m31s Normal Created pod/maas-api-revoke-sub-25dd93bf-8zbsp Container created 2m31s Normal SuccessfulCreate job/maas-api-revoke-sub-40e8492f Created pod: maas-api-revoke-sub-40e8492f-z5ngh 2m31s Normal Pulled pod/maas-api-revoke-sub-25dd93bf-8zbsp Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m30s Normal Pulled pod/maas-api-key-cleanup-29896185-6smnz Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m30s Normal Created pod/maas-api-key-cleanup-29896185-6smnz Container created 2m30s Normal Started pod/maas-api-revoke-sub-40e8492f-z5ngh Container started 2m30s Normal Created pod/maas-api-revoke-sub-40e8492f-z5ngh Container created 2m30s Normal Started pod/maas-api-key-cleanup-29896185-6smnz Container started 2m29s Normal SuccessfulCreate job/maas-api-revoke-sub-3d8b0bd4 Created pod: maas-api-revoke-sub-3d8b0bd4-r84p7 2m29s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m29s Normal Created pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Container created 2m29s Normal Started pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Container started 2m29s Warning BackOff pod/maas-api-key-cleanup-29896185-6smnz Back-off restarting failed container cleanup in pod maas-api-key-cleanup-29896185-6smnz_odh-ai-gateway-infra(306ffc68-1d7d-4ceb-8213-b88dd0ae4ca2) 2m28s Normal SuccessfulDelete job/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185 Deleted pod: maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx 2m28s Warning BackOff pod/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx Back-off restarting failed container cleanup in pod maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185-xd8jx_odh-ai-gateway-infra(fa84199d-f00a-43e8-ba40-077b8d38bdef) 2m28s Normal Created pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Container created 2m28s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m28s Normal Started pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Container started 2m28s Normal SuccessfulDelete job/maas-api-key-cleanup-29896185 Deleted pod: maas-api-key-cleanup-29896185-6smnz 2m28s Normal AddedInterface pod/maas-api-revoke-sub-3d8b0bd4-r84p7 Add eth0 [10.132.0.94/23] from ovn-kubernetes 2m28s Normal Started pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Container started 2m28s Normal Created pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Container created 2m28s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m28s Normal Pulled pod/maas-api-revoke-sub-3d8b0bd4-r84p7 Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m27s Warning BackOff pod/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq Back-off restarting failed container cleanup in pod maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq_odh-ai-gateway-infra(e9cc36a9-c2cf-4734-97f9-fd1f955e2a6d) 2m27s Normal Created pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Container created 2m27s Normal Started pod/maas-api-revoke-sub-3d8b0bd4-r84p7 Container started 2m27s Normal Pulled pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m27s Normal Started pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Container started 2m27s Normal Completed job/maas-api-revoke-sub-25dd93bf Job completed 2m27s Normal Completed job/maas-api-revoke-sub-40e8492f Job completed 2m27s Normal Completed job/maas-api-revoke-sub-ab03cae5 Job completed 2m27s Warning BackOff pod/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg Back-off restarting failed container cleanup in pod maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg_odh-ai-gateway-infra(f58fb3b0-0da4-41c2-b31f-e937ce1daab7) 2m27s Normal Created pod/maas-api-revoke-sub-3d8b0bd4-r84p7 Container created 2m26s Warning BackOff pod/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q Back-off restarting failed container cleanup in pod maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q_odh-ai-gateway-infra(9d35ca3b-adcc-4c9a-bc02-ad6a17ec9390) 2m26s Warning BackoffLimitExceeded job/maas-api-key-cleanup-29896185 Job has reached the specified backoff limit 2m26s Normal SuccessfulDelete job/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185 Deleted pod: maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185-whwdq 2m26s Normal SuccessfulDelete job/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185 Deleted pod: maas-api-key-cleanup-e2e-worker-w0-27935d-29896185-7pslg 2m26s Normal SawCompletedJob cronjob/maas-api-key-cleanup Saw completed job: maas-api-key-cleanup-29896185, condition: Failed 2m25s Normal SawCompletedJob cronjob/maas-api-key-cleanup-e2e-worker-w5-9d808e Saw completed job: maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185, condition: Failed 2m25s Warning BackoffLimitExceeded job/maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185 Job has reached the specified backoff limit 2m25s Normal SuccessfulDelete job/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185 Deleted pod: maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185-2hd8q 2m25s Normal SawCompletedJob cronjob/maas-api-key-cleanup-e2e-worker-w6-2b14d0 Saw completed job: maas-api-key-cleanup-e2e-worker-w6-2b14d0-29896185, condition: Failed 2m25s Warning BackoffLimitExceeded job/maas-api-key-cleanup-e2e-worker-w5-9d808e-29896185 Job has reached the specified backoff limit 2m24s Normal Completed job/maas-api-revoke-sub-3d8b0bd4 Job completed 2m24s Warning BackoffLimitExceeded job/maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185 Job has reached the specified backoff limit 2m24s Normal SawCompletedJob cronjob/maas-api-key-cleanup-e2e-worker-w0-27935d Saw completed job: maas-api-key-cleanup-e2e-worker-w0-27935d-29896185, condition: Failed 2m24s Warning BackoffLimitExceeded job/maas-api-key-cleanup-e2e-worker-w0-27935d-29896185 Job has reached the specified backoff limit 2m24s Normal SawCompletedJob cronjob/maas-api-key-cleanup-e2e-worker-w3-f910f5 Saw completed job: maas-api-key-cleanup-e2e-worker-w3-f910f5-29896185, condition: Failed 2m17s Normal SuccessfulCreate job/maas-api-revoke-sub-e76d4112 Created pod: maas-api-revoke-sub-e76d4112-zgmv6 2m16s Normal SuccessfulCreate job/maas-api-revoke-sub-93ee4734 Created pod: maas-api-revoke-sub-93ee4734-c57px 2m15s Normal Pulled pod/maas-api-revoke-sub-93ee4734-c57px Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m15s Normal SuccessfulCreate job/maas-api-revoke-sub-6aeadf02 Created pod: maas-api-revoke-sub-6aeadf02-q2dnz 2m15s Normal AddedInterface pod/maas-api-revoke-sub-93ee4734-c57px Add eth0 [10.133.0.64/23] from ovn-kubernetes 2m14s Normal Killing pod/maas-api-e2e-worker-w6-2b14d0-6749f7ff76-kzdf6 Stopping container maas-api 2m14s Normal Created pod/maas-api-revoke-sub-93ee4734-c57px Container created 2m14s Normal Started pod/maas-api-revoke-sub-93ee4734-c57px Container started 2m13s Normal AddedInterface pod/maas-api-revoke-sub-6aeadf02-q2dnz Add eth0 [10.132.0.99/23] from ovn-kubernetes 2m13s Normal Pulled pod/maas-api-revoke-sub-6aeadf02-q2dnz Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m13s Normal AddedInterface pod/maas-api-revoke-sub-e76d4112-zgmv6 Add eth0 [10.132.0.98/23] from ovn-kubernetes 2m13s Normal Pulled pod/maas-api-revoke-sub-e76d4112-zgmv6 Container image "registry.redhat.io/ubi9/ubi-minimal:9.7" already present on machine and can be accessed by the pod 2m12s Normal Started pod/maas-api-revoke-sub-e76d4112-zgmv6 Container started 2m12s Normal Created pod/maas-api-revoke-sub-e76d4112-zgmv6 Container created 2m12s Normal Created pod/maas-api-revoke-sub-6aeadf02-q2dnz Container created 2m12s Normal Started pod/maas-api-revoke-sub-6aeadf02-q2dnz Container started 2m11s Normal Completed job/maas-api-revoke-sub-93ee4734 Job completed 2m8s Normal Completed job/maas-api-revoke-sub-e76d4112 Job completed 2m8s Normal Completed job/maas-api-revoke-sub-6aeadf02 Job completed 2m4s Normal Killing pod/maas-api-e2e-worker-w5-9d808e-7788c7bbdd-59fzm Stopping container maas-api 2m4s Normal Killing pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Stopping container maas-api 2m4s Normal Killing pod/maas-api-e2e-worker-w0-27935d-54dd4977df-w2n5m Stopping container maas-api 2m1s Warning Unhealthy pod/maas-api-e2e-worker-w3-f910f5-86f6b57569-lrf7l Readiness probe failed: Get "https://10.134.0.44:8443/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)