Events

Time	Namespace	Component	RelatedObject	Reason	Message
20:37:41	openshift-ingress		payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr to ip-10-0-140-217.ec2.internal
20:37:43	openshift-ingress		payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r to ip-10-0-137-173.ec2.internal
20:19:20	openshift-ingress		data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6 to ip-10-0-129-222.ec2.internal
20:26:15	openshift-ingress		payload-pre-processing-76f657cdc4-bzv64	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-76f657cdc4-bzv64 to ip-10-0-137-173.ec2.internal
20:21:56	openshift-ingress		maas-default-gateway-openshift-default-78bb78ff78-nsf7v	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nsf7v to ip-10-0-129-222.ec2.internal
20:19:11	openshift-ingress		kube-auth-proxy-57479bdc89-7jwcp	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-57479bdc89-7jwcp to ip-10-0-140-217.ec2.internal
20:19:12	openshift-ingress		kube-auth-proxy-57479bdc89-twhd8	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-57479bdc89-twhd8 to ip-10-0-137-173.ec2.internal
20:19:28	openshift-ingress		istiod-openshift-gateway-5677ccb49b-pf6fj	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-pf6fj to ip-10-0-129-222.ec2.internal
20:27:58	openshift-ingress		e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Scheduled	Successfully assigned openshift-ingress/e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck to ip-10-0-137-173.ec2.internal
20:19:15	openshift-ingress		istiod-openshift-gateway-7b7995d5cd-xv956	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xv956 to ip-10-0-137-173.ec2.internal
20:23:18	openshift-ingress		payload-pre-processing-59fbc47bf8-5hnbl	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-59fbc47bf8-5hnbl to ip-10-0-137-173.ec2.internal
20:14:16	openshift-ingress		router-default-69f79489c8-d7hqv	Scheduled	Successfully assigned openshift-ingress/router-default-69f79489c8-d7hqv to ip-10-0-137-173.ec2.internal
20:37:45	openshift-ingress		payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt to ip-10-0-140-217.ec2.internal
20:37:43	openshift-ingress		payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c to ip-10-0-140-217.ec2.internal
20:23:18	openshift-ingress		payload-processing-5b54b7fbb7-br427	Scheduled	Successfully assigned openshift-ingress/payload-processing-5b54b7fbb7-br427 to ip-10-0-137-173.ec2.internal
20:27:58	openshift-ingress		e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	Scheduled	Successfully assigned openshift-ingress/e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9 to ip-10-0-129-222.ec2.internal
20:27:58	openshift-ingress		e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	Scheduled	Successfully assigned openshift-ingress/e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2 to ip-10-0-129-222.ec2.internal
20:26:15	openshift-ingress		payload-processing-85fd65c94c-776s2	Scheduled	Successfully assigned openshift-ingress/payload-processing-85fd65c94c-776s2 to ip-10-0-137-173.ec2.internal
20:37:45	openshift-ingress		payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll to ip-10-0-129-222.ec2.internal
20:37:41	openshift-ingress		payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh to ip-10-0-137-173.ec2.internal
20:08:16	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-69f79489c8 from 0 to 1
20:08:19 (x10)	openshift-ingress	replicaset-controller	router-default-69f79489c8	FailedCreate	Error creating: pods "router-default-69f79489c8-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
20:08:31	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
20:08:31	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
20:08:36	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
20:11:31 (x16)	openshift-ingress	replicaset-controller	router-default-69f79489c8	FailedCreate	Error creating: pods "router-default-69f79489c8-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
20:13:06 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
20:14:16	openshift-ingress	replicaset-controller	router-default-69f79489c8	SuccessfulCreate	Created pod: router-default-69f79489c8-d7hqv
20:14:23 (x5)	openshift-ingress	kubelet	router-default-69f79489c8-d7hqv	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
20:14:31 (x6)	openshift-ingress	kubelet	router-default-69f79489c8-d7hqv	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
20:14:48	openshift-ingress	kubelet	router-default-69f79489c8-d7hqv	Started	Started container router
20:14:48	openshift-ingress	kubelet	router-default-69f79489c8-d7hqv	Created	Created container: router
20:14:48	openshift-ingress	multus	router-default-69f79489c8-d7hqv	AddedInterface	Add eth0 [10.133.0.10/23] from ovn-kubernetes
20:14:48	openshift-ingress	kubelet	router-default-69f79489c8-d7hqv	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f" already present on machine
20:19:11 (x2)	openshift-ingress	replicaset-controller	kube-auth-proxy-57479bdc89	FailedCreate	Error creating: pods "kube-auth-proxy-57479bdc89-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found
20:19:11	openshift-ingress	replicaset-controller	kube-auth-proxy-57479bdc89	SuccessfulCreate	Created pod: kube-auth-proxy-57479bdc89-7jwcp
20:19:11	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-57479bdc89 from 0 to 2
20:19:11	openshift-ingress	replicaset-controller	kube-auth-proxy-57479bdc89	SuccessfulCreate	Created pod: kube-auth-proxy-57479bdc89-twhd8
20:19:12	openshift-ingress	multus	kube-auth-proxy-57479bdc89-7jwcp	AddedInterface	Add eth0 [10.134.0.17/23] from ovn-kubernetes
20:19:12	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-7jwcp	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"
20:19:12	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-twhd8	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"
20:19:12	openshift-ingress	multus	kube-auth-proxy-57479bdc89-twhd8	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
20:19:15	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-twhd8	Started	Started container kube-auth-proxy
20:19:15	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-twhd8	Created	Created container: kube-auth-proxy
20:19:15	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-twhd8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.728s (2.728s including waiting). Image size: 265120132 bytes.
20:19:15	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-7jwcp	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 3.291s (3.291s including waiting). Image size: 265120132 bytes.
20:19:15	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1
20:19:15	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7b7995d5cd	SuccessfulCreate	Created pod: istiod-openshift-gateway-7b7995d5cd-xv956
20:19:16	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-7jwcp	Created	Created container: kube-auth-proxy
20:19:16	openshift-ingress	kubelet	kube-auth-proxy-57479bdc89-7jwcp	Started	Started container kube-auth-proxy
20:19:16	openshift-ingress	multus	istiod-openshift-gateway-7b7995d5cd-xv956	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
20:19:16	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-xv956	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"
20:19:18	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-xv956	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.421s (2.421s including waiting). Image size: 181126104 bytes.
20:19:19	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-xv956	Created	Created container: discovery
20:19:19	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-xv956	Started	Started container discovery
20:19:20	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-d66cd8c7c	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6
20:19:20	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-d66cd8c7c from 0 to 1
20:19:21	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"
20:19:21	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
20:19:26	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	Started	Started container istio-proxy
20:19:26	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 4.655s (4.655s including waiting). Image size: 181318953 bytes.
20:19:26	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-d66cd8c7c-mp4n6	Created	Created container: istio-proxy
20:19:28	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1
20:19:28	openshift-ingress	replicaset-controller	istiod-openshift-gateway-5677ccb49b	SuccessfulCreate	Created pod: istiod-openshift-gateway-5677ccb49b-pf6fj
20:19:29	openshift-ingress	multus	istiod-openshift-gateway-5677ccb49b-pf6fj	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
20:19:29	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"
20:19:31	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.357s (2.357s including waiting). Image size: 181126104 bytes.
20:19:31	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	Created	Created container: discovery
20:19:31	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	Started	Started container discovery
20:19:32	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:19:32	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-pf6fj	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:19:35	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0
20:19:35	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7b7995d5cd	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7b7995d5cd-xv956
20:19:35	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-xv956	Killing	Stopping container discovery
20:19:41 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:41 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:45 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:45 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:00	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:20:00	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:21:56	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
20:21:56	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-78bb78ff78 from 0 to 1
20:21:56	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-78bb78ff78	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-78bb78ff78-nsf7v
20:21:57	openshift-ingress	multus	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
20:23:18 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-59fbc47bf8	FailedCreate	Error creating: pods "payload-pre-processing-59fbc47bf8-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:23:18	openshift-ingress	replicaset-controller	payload-pre-processing-59fbc47bf8	SuccessfulCreate	Created pod: payload-pre-processing-59fbc47bf8-5hnbl
20:23:18 (x6)	openshift-ingress	replicaset-controller	payload-processing-5b54b7fbb7	FailedCreate	Error creating: pods "payload-processing-5b54b7fbb7-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:23:18	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-59fbc47bf8 from 0 to 1
20:23:18	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5b54b7fbb7 from 0 to 1
20:23:18	openshift-ingress	replicaset-controller	payload-processing-5b54b7fbb7	SuccessfulCreate	Created pod: payload-processing-5b54b7fbb7-br427
20:23:19	openshift-ingress	multus	payload-pre-processing-59fbc47bf8-5hnbl	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
20:23:19	openshift-ingress	kubelet	payload-pre-processing-59fbc47bf8-5hnbl	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
20:23:19	openshift-ingress	multus	payload-processing-5b54b7fbb7-br427	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
20:23:19	openshift-ingress	kubelet	payload-processing-5b54b7fbb7-br427	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
20:23:20	openshift-ingress	kubelet	payload-pre-processing-59fbc47bf8-5hnbl	Started	Started container payload-pre-processing
20:23:20	openshift-ingress	kubelet	payload-pre-processing-59fbc47bf8-5hnbl	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.303s (1.303s including waiting). Image size: 166453408 bytes.
20:23:20	openshift-ingress	kubelet	payload-pre-processing-59fbc47bf8-5hnbl	Created	Created container: payload-pre-processing
20:23:20	openshift-ingress	kubelet	payload-processing-5b54b7fbb7-br427	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 878ms (878ms including waiting). Image size: 166453408 bytes.
20:23:20	openshift-ingress	kubelet	payload-processing-5b54b7fbb7-br427	Started	Started container payload-processing
20:23:20	openshift-ingress	kubelet	payload-processing-5b54b7fbb7-br427	Created	Created container: payload-processing
20:23:27 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:23:28 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	Started	Started container istio-proxy
20:23:28 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	Created	Created container: istio-proxy
20:23:33 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	Unhealthy	Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused
20:23:33 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-78bb78ff78-nsf7v	ProbeError	Startup probe error: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused body:
20:25:53	openshift-ingress	kubelet	payload-processing-5b54b7fbb7-br427	Killing	Stopping container payload-processing
20:25:53	openshift-ingress	kubelet	payload-pre-processing-59fbc47bf8-5hnbl	Killing	Stopping container payload-pre-processing
20:26:15 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-76f657cdc4	FailedCreate	Error creating: pods "payload-pre-processing-76f657cdc4-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:26:15	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-76f657cdc4 from 0 to 1
20:26:15	openshift-ingress	replicaset-controller	payload-pre-processing-76f657cdc4	SuccessfulCreate	Created pod: payload-pre-processing-76f657cdc4-bzv64
20:26:15	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-85fd65c94c from 0 to 1
20:26:15 (x5)	openshift-ingress	replicaset-controller	payload-processing-85fd65c94c	FailedCreate	Error creating: pods "payload-processing-85fd65c94c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:26:15	openshift-ingress	replicaset-controller	payload-processing-85fd65c94c	SuccessfulCreate	Created pod: payload-processing-85fd65c94c-776s2
20:26:16	openshift-ingress	multus	payload-processing-85fd65c94c-776s2	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
20:26:16	openshift-ingress	kubelet	payload-processing-85fd65c94c-776s2	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:26:16	openshift-ingress	kubelet	payload-pre-processing-76f657cdc4-bzv64	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:26:16	openshift-ingress	multus	payload-pre-processing-76f657cdc4-bzv64	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
20:26:17	openshift-ingress	kubelet	payload-processing-85fd65c94c-776s2	Started	Started container payload-processing
20:26:17	openshift-ingress	kubelet	payload-processing-85fd65c94c-776s2	Created	Created container: payload-processing
20:26:17	openshift-ingress	kubelet	payload-pre-processing-76f657cdc4-bzv64	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.104s (1.104s including waiting). Image size: 141049838 bytes.
20:26:17	openshift-ingress	kubelet	payload-pre-processing-76f657cdc4-bzv64	Created	Created container: payload-pre-processing
20:26:17	openshift-ingress	kubelet	payload-pre-processing-76f657cdc4-bzv64	Started	Started container payload-pre-processing
20:26:17	openshift-ingress	kubelet	payload-processing-85fd65c94c-776s2	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.041s (1.041s including waiting). Image size: 141049838 bytes.
20:27:34	openshift-ingress	gateway_labeler_controller	e2e-inf-a-26ffff	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-26ffff
20:27:34	openshift-ingress	gateway_labeler_controller	e2e-worker-w3-74bb52	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-74bb52
20:27:34	openshift-ingress	gateway_labeler_controller	e2e-worker-w0-6ddb93	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-6ddb93
20:27:58	openshift-ingress	deployment-controller	e2e-worker-w0-6ddb93-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-worker-w0-6ddb93-openshift-default-c4d89d488 from 0 to 1
20:27:58	openshift-ingress	replicaset-controller	e2e-worker-w3-74bb52-openshift-default-7747fb58b6	SuccessfulCreate	Created pod: e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9
20:27:58	openshift-ingress	replicaset-controller	e2e-inf-a-26ffff-openshift-default-55668cd47d	SuccessfulCreate	Created pod: e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck
20:27:58	openshift-ingress	deployment-controller	e2e-inf-a-26ffff-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-inf-a-26ffff-openshift-default-55668cd47d from 0 to 1
20:27:58	openshift-ingress	deployment-controller	e2e-worker-w3-74bb52-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-worker-w3-74bb52-openshift-default-7747fb58b6 from 0 to 1
20:27:58	openshift-ingress	replicaset-controller	e2e-worker-w0-6ddb93-openshift-default-c4d89d488	SuccessfulCreate	Created pod: e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2
20:27:59	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"
20:27:59	openshift-ingress	multus	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
20:27:59	openshift-ingress	multus	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
20:27:59	openshift-ingress	multus	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
20:28:02	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.743s (2.743s including waiting). Image size: 181318953 bytes.
20:37:40	openshift-ingress	deployment-controller	payload-pre-processing-e2e-worker-w0-6ddb93	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b from 0 to 1
20:37:40	openshift-ingress	deployment-controller	payload-processing-e2e-worker-w0-6ddb93	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f from 0 to 1
20:37:41	openshift-ingress	replicaset-controller	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f	SuccessfulCreate	Created pod: payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh
20:37:41	openshift-ingress	multus	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
20:37:41 (x6)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b	FailedCreate	Error creating: pods "payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6ddb93: serviceaccount "payload-processing-e2e-worker-w0-6ddb93" not found
20:37:41	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b	SuccessfulCreate	Created pod: payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr
20:37:41 (x7)	openshift-ingress	replicaset-controller	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f	FailedCreate	Error creating: pods "payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6ddb93: serviceaccount "payload-processing-e2e-worker-w0-6ddb93" not found
20:37:42	openshift-ingress	kubelet	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh	Started	Started container payload-processing
20:37:42	openshift-ingress	deployment-controller	payload-processing-e2e-worker-w3-74bb52	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-worker-w3-74bb52-65f96b54fc from 0 to 1
20:37:42	openshift-ingress	multus	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
20:37:42	openshift-ingress	kubelet	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:37:42	openshift-ingress	kubelet	payload-processing-e2e-worker-w0-6ddb93-5bbc557c9f-jhhhh	Created	Created container: payload-processing
20:37:42	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
20:37:43	openshift-ingress	replicaset-controller	payload-processing-e2e-worker-w3-74bb52-65f96b54fc	SuccessfulCreate	Created pod: payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c
20:37:43 (x6)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d	FailedCreate	Error creating: pods "payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-74bb52: serviceaccount "payload-processing-e2e-worker-w3-74bb52" not found
20:37:43	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d	SuccessfulCreate	Created pod: payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r
20:37:43	openshift-ingress	deployment-controller	payload-pre-processing-e2e-worker-w3-74bb52	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d from 0 to 1
20:37:43 (x7)	openshift-ingress	replicaset-controller	payload-processing-e2e-worker-w3-74bb52-65f96b54fc	FailedCreate	Error creating: pods "payload-processing-e2e-worker-w3-74bb52-65f96b54fc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-74bb52: serviceaccount "payload-processing-e2e-worker-w3-74bb52" not found
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	Started	Started container payload-pre-processing
20:37:44	openshift-ingress	multus	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
20:37:44	openshift-ingress	kubelet	payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c	Started	Started container payload-processing
20:37:44	openshift-ingress	kubelet	payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c	Created	Created container: payload-processing
20:37:44	openshift-ingress	kubelet	payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:37:44	openshift-ingress	multus	payload-processing-e2e-worker-w3-74bb52-65f96b54fc-f4x7c	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r	Started	Started container payload-pre-processing
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.235s (2.235s including waiting). Image size: 166453408 bytes.
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w0-6ddb93-865b7bf4b-7qlrr	Created	Created container: payload-pre-processing
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r	Created	Created container: payload-pre-processing
20:37:44	openshift-ingress	kubelet	payload-pre-processing-e2e-worker-w3-74bb52-6b7b75dd9d-gx86r	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:37:45 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-a-26ffff-6798bd66	FailedCreate	Error creating: pods "payload-pre-processing-e2e-inf-a-26ffff-6798bd66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-26ffff: serviceaccount "payload-processing-e2e-inf-a-26ffff" not found
20:37:45 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-a-26ffff-6f46768669	FailedCreate	Error creating: pods "payload-processing-e2e-inf-a-26ffff-6f46768669-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-26ffff: serviceaccount "payload-processing-e2e-inf-a-26ffff" not found
20:37:45	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-a-26ffff-6798bd66	SuccessfulCreate	Created pod: payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt
20:37:45	openshift-ingress	deployment-controller	payload-processing-e2e-inf-a-26ffff	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-inf-a-26ffff-6f46768669 from 0 to 1
20:37:45	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-a-26ffff-6f46768669	SuccessfulCreate	Created pod: payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll
20:37:45	openshift-ingress	deployment-controller	payload-pre-processing-e2e-inf-a-26ffff	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-inf-a-26ffff-6798bd66 from 0 to 1
20:37:46	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:37:46	openshift-ingress	multus	payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
20:37:46	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt	Created	Created container: payload-pre-processing
20:37:46	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-26ffff-6798bd66-wjxjt	Started	Started container payload-pre-processing
20:37:46	openshift-ingress	kubelet	payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
20:37:46	openshift-ingress	multus	payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
20:37:48	openshift-ingress	kubelet	payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	Started	Started container payload-processing
20:37:48	openshift-ingress	kubelet	payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	Created	Created container: payload-processing
20:37:48	openshift-ingress	kubelet	payload-processing-e2e-inf-a-26ffff-6f46768669-hgqll	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.873s (1.873s including waiting). Image size: 166453408 bytes.
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	Started	Started container istio-proxy
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	Created	Created container: istio-proxy
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	Started	Started container istio-proxy
20:37:55 (x2)	openshift-ingress	kubelet	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	Created	Created container: istio-proxy
20:38:01 (x5)	openshift-ingress	kubelet	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	ProbeError	Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused body:
20:38:01 (x5)	openshift-ingress	kubelet	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	Unhealthy	Startup probe failed: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused
20:38:01 (x5)	openshift-ingress	kubelet	e2e-worker-w3-74bb52-openshift-default-7747fb58b6-kg6g9	ProbeError	Startup probe error: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused body:
20:38:01 (x5)	openshift-ingress	kubelet	e2e-worker-w0-6ddb93-openshift-default-c4d89d488-zwbx2	Unhealthy	Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused
20:38:03	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:38:03 (x2)	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Started	Started container istio-proxy
20:38:03 (x2)	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Created	Created container: istio-proxy
20:38:09 (x5)	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	Unhealthy	Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused
20:38:09 (x5)	openshift-ingress	kubelet	e2e-inf-a-26ffff-openshift-default-55668cd47d-rv2ck	ProbeError	Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused body:
20:40:12	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
20:40:13	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer