Events

Time	Namespace	Component	RelatedObject	Reason	Message
20:16:39	openshift-ingress		payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2 to ip-10-0-129-235.ec2.internal
20:06:54	openshift-ingress		payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw to ip-10-0-129-235.ec2.internal
19:33:59	openshift-ingress		kube-auth-proxy-5b7cdcf968-mgjtv	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-5b7cdcf968-mgjtv to ip-10-0-128-237.ec2.internal
19:59:35	openshift-ingress		e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Scheduled	Successfully assigned openshift-ingress/e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw to ip-10-0-142-24.ec2.internal
19:33:59	openshift-ingress		kube-auth-proxy-5b7cdcf968-xsw47	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-5b7cdcf968-xsw47 to ip-10-0-129-235.ec2.internal
20:00:28	openshift-ingress		e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Scheduled	Successfully assigned openshift-ingress/e2e-mt-73760660-openshift-default-6dc79f5684-95xgl to ip-10-0-142-24.ec2.internal
19:36:28	openshift-ingress		maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-gxwfr to ip-10-0-128-237.ec2.internal
19:56:00	openshift-ingress		e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Scheduled	Successfully assigned openshift-ingress/e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f to ip-10-0-128-237.ec2.internal
19:56:38	openshift-ingress		e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Scheduled	Successfully assigned openshift-ingress/e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5 to ip-10-0-142-24.ec2.internal
19:37:52	openshift-ingress		payload-pre-processing-687dbb6cf6-ct46q	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-687dbb6cf6-ct46q to ip-10-0-128-237.ec2.internal
20:17:50	openshift-ingress		e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Scheduled	Successfully assigned openshift-ingress/e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m to ip-10-0-128-237.ec2.internal
19:56:42	openshift-ingress		payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r to ip-10-0-129-235.ec2.internal
19:55:23	openshift-ingress		payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd to ip-10-0-128-237.ec2.internal
20:16:35	openshift-ingress		e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Scheduled	Successfully assigned openshift-ingress/e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5 to ip-10-0-142-24.ec2.internal
19:59:39	openshift-ingress		payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq to ip-10-0-129-235.ec2.internal
20:05:44	openshift-ingress		payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb to ip-10-0-129-235.ec2.internal
20:05:52	openshift-ingress		payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss to ip-10-0-129-235.ec2.internal
19:57:22	openshift-ingress		payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5 to ip-10-0-129-235.ec2.internal
20:03:26	openshift-ingress		payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84 to ip-10-0-129-235.ec2.internal
20:03:35	openshift-ingress		payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8 to ip-10-0-129-235.ec2.internal
20:16:21	openshift-ingress		payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h to ip-10-0-129-235.ec2.internal
19:55:20	openshift-ingress		e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Scheduled	Successfully assigned openshift-ingress/e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm to ip-10-0-142-24.ec2.internal
20:16:39	openshift-ingress		payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259 to ip-10-0-129-235.ec2.internal
20:17:54	openshift-ingress		payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj to ip-10-0-129-235.ec2.internal
19:56:03	openshift-ingress		payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w to ip-10-0-129-235.ec2.internal
20:00:31	openshift-ingress		payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6 to ip-10-0-129-235.ec2.internal
20:16:17	openshift-ingress		e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Scheduled	Successfully assigned openshift-ingress/e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh to ip-10-0-128-237.ec2.internal
20:06:50	openshift-ingress		e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5 to ip-10-0-142-24.ec2.internal
19:34:00	openshift-ingress		istiod-openshift-gateway-7b7995d5cd-clh22	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-clh22 to ip-10-0-128-237.ec2.internal
19:34:06	openshift-ingress		data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c58mpl to ip-10-0-142-24.ec2.internal
20:07:03	openshift-ingress		payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j to ip-10-0-129-235.ec2.internal
20:03:30	openshift-ingress		e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Scheduled	Successfully assigned openshift-ingress/e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9 to ip-10-0-128-237.ec2.internal
20:05:40	openshift-ingress		e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Scheduled	Successfully assigned openshift-ingress/e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw to ip-10-0-142-24.ec2.internal
19:37:52	openshift-ingress		payload-processing-5cc6cd444b-gbhk5	Scheduled	Successfully assigned openshift-ingress/payload-processing-5cc6cd444b-gbhk5 to ip-10-0-128-237.ec2.internal
19:56:42	openshift-ingress		payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf to ip-10-0-129-235.ec2.internal
19:55:23	openshift-ingress		payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph to ip-10-0-128-237.ec2.internal
20:03:22	openshift-ingress		e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Scheduled	Successfully assigned openshift-ingress/e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw to ip-10-0-142-24.ec2.internal
19:29:23	openshift-ingress		router-default-645bb56f9f-b8xdb	Scheduled	Successfully assigned openshift-ingress/router-default-645bb56f9f-b8xdb to ip-10-0-128-237.ec2.internal
19:27:28	openshift-ingress		router-default-645bb56f9f-b8xdb	FailedScheduling	0/3 nodes are available: 3 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
20:05:44	openshift-ingress		payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g to ip-10-0-129-235.ec2.internal
20:07:03	openshift-ingress		payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb to ip-10-0-129-235.ec2.internal
20:06:54	openshift-ingress		payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d to ip-10-0-129-235.ec2.internal
19:59:39	openshift-ingress		payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44 to ip-10-0-129-235.ec2.internal
20:00:31	openshift-ingress		payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-mt-73760660-8ddd65d58-h9p57 to ip-10-0-129-235.ec2.internal
20:06:58	openshift-ingress		e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg to ip-10-0-128-237.ec2.internal
20:05:48	openshift-ingress		e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Scheduled	Successfully assigned openshift-ingress/e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg to ip-10-0-128-237.ec2.internal
20:05:52	openshift-ingress		payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q to ip-10-0-129-235.ec2.internal
19:34:08	openshift-ingress		istiod-openshift-gateway-5677ccb49b-mp89s	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-mp89s to ip-10-0-142-24.ec2.internal
19:57:22	openshift-ingress		payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4 to ip-10-0-129-235.ec2.internal
20:03:26	openshift-ingress		payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-79961d-79f89db66-88g6l to ip-10-0-129-235.ec2.internal
19:57:19	openshift-ingress		e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Scheduled	Successfully assigned openshift-ingress/e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw to ip-10-0-128-237.ec2.internal
20:03:35	openshift-ingress		payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8 to ip-10-0-129-235.ec2.internal
20:16:21	openshift-ingress		payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-018406-75754c7976-84hjb to ip-10-0-129-235.ec2.internal
19:56:03	openshift-ingress		payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g to ip-10-0-129-235.ec2.internal
20:17:54	openshift-ingress		payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6 to ip-10-0-129-235.ec2.internal
19:21:15	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-645bb56f9f from 0 to 1
19:21:25 (x12)	openshift-ingress	replicaset-controller	router-default-645bb56f9f	FailedCreate	Error creating: pods "router-default-645bb56f9f-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
19:21:51	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
19:21:51	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
19:21:56	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
19:24:33 (x16)	openshift-ingress	replicaset-controller	router-default-645bb56f9f	FailedCreate	Error creating: pods "router-default-645bb56f9f-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
19:27:08 (x13)	openshift-ingress	replicaset-controller	router-default-645bb56f9f	FailedCreate	Error creating: pods "router-default-645bb56f9f-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
19:27:28	openshift-ingress	replicaset-controller	router-default-645bb56f9f	SuccessfulCreate	Created pod: router-default-645bb56f9f-b8xdb
19:29:23	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
19:29:25	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
19:29:39 (x6)	openshift-ingress	kubelet	router-default-645bb56f9f-b8xdb	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
19:29:39 (x6)	openshift-ingress	kubelet	router-default-645bb56f9f-b8xdb	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
19:29:56	openshift-ingress	kubelet	router-default-645bb56f9f-b8xdb	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559" already present on machine
19:29:56	openshift-ingress	kubelet	router-default-645bb56f9f-b8xdb	Created	Created container: router
19:29:56	openshift-ingress	kubelet	router-default-645bb56f9f-b8xdb	Started	Started container router
19:29:56	openshift-ingress	multus	router-default-645bb56f9f-b8xdb	AddedInterface	Add eth0 [10.132.0.21/23] from ovn-kubernetes
19:33:59	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-xsw47	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"
19:33:59	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-5b7cdcf968 from 0 to 2
19:33:59	openshift-ingress	multus	kube-auth-proxy-5b7cdcf968-xsw47	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
19:33:59 (x5)	openshift-ingress	replicaset-controller	kube-auth-proxy-5b7cdcf968	FailedCreate	Error creating: pods "kube-auth-proxy-5b7cdcf968-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found
19:33:59	openshift-ingress	replicaset-controller	kube-auth-proxy-5b7cdcf968	SuccessfulCreate	Created pod: kube-auth-proxy-5b7cdcf968-mgjtv
19:33:59	openshift-ingress	replicaset-controller	kube-auth-proxy-5b7cdcf968	SuccessfulCreate	Created pod: kube-auth-proxy-5b7cdcf968-xsw47
19:33:59	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-mgjtv	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"
19:33:59	openshift-ingress	multus	kube-auth-proxy-5b7cdcf968-mgjtv	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
19:34:00	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7b7995d5cd	SuccessfulCreate	Created pod: istiod-openshift-gateway-7b7995d5cd-clh22
19:34:00	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1
19:34:01	openshift-ingress	multus	istiod-openshift-gateway-7b7995d5cd-clh22	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
19:34:01	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-xsw47	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.009s (2.009s including waiting). Image size: 265120132 bytes.
19:34:01	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-xsw47	Started	Started container kube-auth-proxy
19:34:01	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-clh22	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"
19:34:01	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-xsw47	Created	Created container: kube-auth-proxy
19:34:03	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-mgjtv	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 3.786s (3.787s including waiting). Image size: 265120132 bytes.
19:34:04	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-mgjtv	Started	Started container kube-auth-proxy
19:34:04	openshift-ingress	kubelet	kube-auth-proxy-5b7cdcf968-mgjtv	Created	Created container: kube-auth-proxy
19:34:05	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-clh22	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 3.897s (3.897s including waiting). Image size: 181126104 bytes.
19:34:05	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-clh22	Created	Created container: discovery
19:34:05	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-68cf4d799c from 0 to 1
19:34:05	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-clh22	Started	Started container discovery
19:34:06	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	AddedInterface	Add eth0 [10.134.0.20/23] from ovn-kubernetes
19:34:06	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"
19:34:06	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-68cf4d799c	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-68cf4d799c58mpl
19:34:08	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1
19:34:08	openshift-ingress	replicaset-controller	istiod-openshift-gateway-5677ccb49b	SuccessfulCreate	Created pod: istiod-openshift-gateway-5677ccb49b-mp89s
19:34:09	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-mp89s	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"
19:34:09	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.636s (2.636s including waiting). Image size: 181318953 bytes.
19:34:09	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	Created	Created container: istio-proxy
19:34:09	openshift-ingress	multus	istiod-openshift-gateway-5677ccb49b-mp89s	AddedInterface	Add eth0 [10.134.0.21/23] from ovn-kubernetes
19:34:09	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-68cf4d799c58mpl	Started	Started container istio-proxy
19:34:12	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-mp89s	Created	Created container: discovery
19:34:12	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-mp89s	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.738s (2.738s including waiting). Image size: 181126104 bytes.
19:34:12	openshift-ingress	kubelet	istiod-openshift-gateway-5677ccb49b-mp89s	Started	Started container discovery
19:34:13	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7b7995d5cd	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7b7995d5cd-clh22
19:34:13	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0
19:34:13	openshift-ingress	kubelet	istiod-openshift-gateway-7b7995d5cd-clh22	Killing	Stopping container discovery
19:34:14	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:14	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:29	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:29	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:45 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:45 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:28	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
19:36:28	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"
19:36:28	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-5ccb968c66	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-5ccb968c66-gxwfr
19:36:28	openshift-ingress	multus	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
19:36:28	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-5ccb968c66 from 0 to 1
19:36:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.654s (2.654s including waiting). Image size: 181318953 bytes.
19:37:51	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5cc6cd444b from 0 to 1
19:37:52 (x5)	openshift-ingress	replicaset-controller	payload-processing-5cc6cd444b	FailedCreate	Error creating: pods "payload-processing-5cc6cd444b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
19:37:52	openshift-ingress	multus	payload-processing-5cc6cd444b-gbhk5	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:37:52	openshift-ingress	multus	payload-pre-processing-687dbb6cf6-ct46q	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:37:52	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-687dbb6cf6 from 0 to 1
19:37:52	openshift-ingress	kubelet	payload-pre-processing-687dbb6cf6-ct46q	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
19:37:52	openshift-ingress	kubelet	payload-processing-5cc6cd444b-gbhk5	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
19:37:52 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-687dbb6cf6	FailedCreate	Error creating: pods "payload-pre-processing-687dbb6cf6-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
19:37:52	openshift-ingress	replicaset-controller	payload-pre-processing-687dbb6cf6	SuccessfulCreate	Created pod: payload-pre-processing-687dbb6cf6-ct46q
19:37:52	openshift-ingress	replicaset-controller	payload-processing-5cc6cd444b	SuccessfulCreate	Created pod: payload-processing-5cc6cd444b-gbhk5
19:37:56	openshift-ingress	kubelet	payload-processing-5cc6cd444b-gbhk5	Created	Created container: payload-processing
19:37:56	openshift-ingress	kubelet	payload-pre-processing-687dbb6cf6-ct46q	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 3.424s (3.424s including waiting). Image size: 165907129 bytes.
19:37:56	openshift-ingress	kubelet	payload-pre-processing-687dbb6cf6-ct46q	Created	Created container: payload-pre-processing
19:37:56	openshift-ingress	kubelet	payload-pre-processing-687dbb6cf6-ct46q	Started	Started container payload-pre-processing
19:37:56	openshift-ingress	kubelet	payload-processing-5cc6cd444b-gbhk5	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 3.39s (3.39s including waiting). Image size: 165907129 bytes.
19:37:56	openshift-ingress	kubelet	payload-processing-5cc6cd444b-gbhk5	Started	Started container payload-processing
19:37:59 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Started	Started container istio-proxy
19:37:59 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Created	Created container: istio-proxy
19:37:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:38:03 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	ProbeError	Startup probe error: Get "http://10.132.0.37:15021/healthz/ready": dial tcp 10.132.0.37:15021: connect: connection refused body:
19:38:03 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-5ccb968c66-gxwfr	Unhealthy	Startup probe failed: Get "http://10.132.0.37:15021/healthz/ready": dial tcp 10.132.0.37:15021: connect: connection refused
19:55:20	openshift-ingress	deployment-controller	e2e-ait-aad6c451-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-ait-aad6c451-openshift-default-5949dc7c54 from 0 to 1
19:55:20	openshift-ingress	gateway_labeler_controller	e2e-ait-aad6c451	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-ait-aad6c451
19:55:20	openshift-ingress	replicaset-controller	e2e-ait-aad6c451-openshift-default-5949dc7c54	SuccessfulCreate	Created pod: e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm
19:55:21	openshift-ingress	multus	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:55:23 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8	FailedCreate	Error creating: pods "payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-aad6c451: serviceaccount "payload-processing-e2e-ait-aad6c451" not found
19:55:23	openshift-ingress	deployment-controller	payload-pre-processing-e2e-ait-aad6c451	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8 from 0 to 1
19:55:23	openshift-ingress	deployment-controller	payload-processing-e2e-ait-aad6c451	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-ait-aad6c451-ddcf4774 from 0 to 1
19:55:23	openshift-ingress	replicaset-controller	payload-processing-e2e-ait-aad6c451-ddcf4774	SuccessfulCreate	Created pod: payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph
19:55:23 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-ait-aad6c451-ddcf4774	FailedCreate	Error creating: pods "payload-processing-e2e-ait-aad6c451-ddcf4774-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-aad6c451: serviceaccount "payload-processing-e2e-ait-aad6c451" not found
19:55:23	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8	SuccessfulCreate	Created pod: payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd
19:55:24	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	Started	Started container payload-pre-processing
19:55:24	openshift-ingress	kubelet	payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	Started	Started container payload-processing
19:55:24	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	Created	Created container: payload-pre-processing
19:55:24	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:55:24	openshift-ingress	kubelet	payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	Created	Created container: payload-processing
19:55:24	openshift-ingress	kubelet	payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:55:24	openshift-ingress	multus	payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
19:55:24	openshift-ingress	multus	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
19:55:34 (x2)	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Created	Created container: istio-proxy
19:55:34 (x2)	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Started	Started container istio-proxy
19:55:34 (x2)	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:55:39 (x4)	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Unhealthy	Startup probe failed: Get "http://10.134.0.25:15021/healthz/ready": dial tcp 10.134.0.25:15021: connect: connection refused
19:55:39 (x4)	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	ProbeError	Startup probe error: Get "http://10.134.0.25:15021/healthz/ready": dial tcp 10.134.0.25:15021: connect: connection refused body:
19:55:58	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-aad6c451-57f65ffdd8-kjbvd	Killing	Stopping container payload-pre-processing
19:55:58	openshift-ingress	kubelet	payload-processing-e2e-ait-aad6c451-ddcf4774-t2zph	Killing	Stopping container payload-processing
19:55:59	openshift-ingress	kubelet	e2e-ait-aad6c451-openshift-default-5949dc7c54-cpmmm	Killing	Stopping container istio-proxy
19:56:00	openshift-ingress	multus	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
19:56:00	openshift-ingress	gateway_labeler_controller	e2e-migrate-f82daff9-gw	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-f82daff9-gw
19:56:00	openshift-ingress	deployment-controller	e2e-migrate-f82daff9-gw-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484 from 0 to 1
19:56:00	openshift-ingress	replicaset-controller	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484	SuccessfulCreate	Created pod: e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f
19:56:03 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-migrate-f82daff9-789c7894b5	FailedCreate	Error creating: pods "payload-processing-e2e-migrate-f82daff9-789c7894b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-f82daff9: serviceaccount "payload-processing-e2e-migrate-f82daff9" not found
19:56:03	openshift-ingress	kubelet	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
19:56:03	openshift-ingress	kubelet	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"
19:56:03	openshift-ingress	deployment-controller	payload-processing-e2e-migrate-f82daff9	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-migrate-f82daff9-789c7894b5 from 0 to 1
19:56:03	openshift-ingress	replicaset-controller	payload-processing-e2e-migrate-f82daff9-789c7894b5	SuccessfulCreate	Created pod: payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g
19:56:03	openshift-ingress	multus	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
19:56:03 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556	FailedCreate	Error creating: pods "payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-f82daff9: serviceaccount "payload-processing-e2e-migrate-f82daff9" not found
19:56:03	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556	SuccessfulCreate	Created pod: payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w
19:56:03	openshift-ingress	deployment-controller	payload-pre-processing-e2e-migrate-f82daff9	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-migrate-f82daff9-d47b5b556 from 0 to 1
19:56:03	openshift-ingress	multus	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
19:56:05	openshift-ingress	kubelet	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Started	Started container payload-pre-processing
19:56:05	openshift-ingress	kubelet	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.842s (1.842s including waiting). Image size: 165907129 bytes.
19:56:05	openshift-ingress	kubelet	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Created	Created container: payload-processing
19:56:05	openshift-ingress	kubelet	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Started	Started container payload-processing
19:56:05	openshift-ingress	kubelet	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Created	Created container: payload-pre-processing
19:56:05	openshift-ingress	kubelet	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.873s (1.873s including waiting). Image size: 165907129 bytes.
19:56:17	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Unhealthy	Readiness probe failed: Get "http://10.132.0.54:15021/healthz/ready": read tcp 10.132.0.2:42982->10.132.0.54:15021: read: connection reset by peer
19:56:17	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	ProbeError	Readiness probe error: Get "http://10.132.0.54:15021/healthz/ready": read tcp 10.132.0.2:42982->10.132.0.54:15021: read: connection reset by peer body:
19:56:18 (x2)	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:56:18 (x2)	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Created	Created container: istio-proxy
19:56:18 (x2)	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Started	Started container istio-proxy
19:56:22 (x4)	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	ProbeError	Startup probe error: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused body:
19:56:22 (x4)	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Unhealthy	Startup probe failed: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused
19:56:37	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-migrate-f82daff9	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-migrate-f82daff9: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-migrate-f82daff9": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-migrate-f82daff9, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9c9e6926-9aeb-493a-bf5a-18bc6c5f12d4, UID in object meta:
19:56:37	openshift-ingress	kubelet	payload-pre-processing-e2e-migrate-f82daff9-d47b5b556-wp58w	Killing	Stopping container payload-pre-processing
19:56:37	openshift-ingress	kubelet	payload-processing-e2e-migrate-f82daff9-789c7894b5-vdk5g	Killing	Stopping container payload-processing
19:56:38	openshift-ingress	deployment-controller	e2e-ait-6e159107-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-ait-6e159107-openshift-default-6ccb5f479b from 0 to 1
19:56:38	openshift-ingress	replicaset-controller	e2e-ait-6e159107-openshift-default-6ccb5f479b	SuccessfulCreate	Created pod: e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5
19:56:38	openshift-ingress	gateway_labeler_controller	e2e-ait-6e159107	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-ait-6e159107
19:56:38	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
19:56:38	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:56:38	openshift-ingress	kubelet	e2e-migrate-f82daff9-gw-openshift-default-5cb7fb7484-89j5f	Killing	Stopping container istio-proxy
19:56:39	openshift-ingress	multus	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:56:41 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ait-6e159107-5867c45f5b	FailedCreate	Error creating: pods "payload-pre-processing-e2e-ait-6e159107-5867c45f5b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-6e159107: serviceaccount "payload-processing-e2e-ait-6e159107" not found
19:56:41	openshift-ingress	deployment-controller	payload-processing-e2e-ait-6e159107	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-ait-6e159107-6dccb7cc84 from 0 to 1
19:56:41 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-ait-6e159107-6dccb7cc84	FailedCreate	Error creating: pods "payload-processing-e2e-ait-6e159107-6dccb7cc84-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-6e159107: serviceaccount "payload-processing-e2e-ait-6e159107" not found
19:56:41	openshift-ingress	deployment-controller	payload-pre-processing-e2e-ait-6e159107	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-ait-6e159107-5867c45f5b from 0 to 1
19:56:42	openshift-ingress	multus	payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
19:56:42	openshift-ingress	kubelet	payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	Created	Created container: payload-processing
19:56:42	openshift-ingress	kubelet	payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:56:42	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	Started	Started container payload-pre-processing
19:56:42	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	Created	Created container: payload-pre-processing
19:56:42	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:56:42	openshift-ingress	multus	payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
19:56:42	openshift-ingress	replicaset-controller	payload-processing-e2e-ait-6e159107-6dccb7cc84	SuccessfulCreate	Created pod: payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf
19:56:42	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ait-6e159107-5867c45f5b	SuccessfulCreate	Created pod: payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r
19:56:42	openshift-ingress	kubelet	payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	Started	Started container payload-processing
19:56:55	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Unhealthy	Readiness probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused
19:56:55 (x2)	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:56:55	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	ProbeError	Readiness probe error: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused body:
19:56:56 (x2)	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Created	Created container: istio-proxy
19:56:56 (x2)	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Started	Started container istio-proxy
19:57:01 (x5)	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Unhealthy	Startup probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused
19:57:01 (x5)	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	ProbeError	Startup probe error: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused body:
19:57:17	openshift-ingress	kubelet	payload-pre-processing-e2e-ait-6e159107-5867c45f5b-pdf5r	Killing	Stopping container payload-pre-processing
19:57:17	openshift-ingress	kubelet	payload-processing-e2e-ait-6e159107-6dccb7cc84-gzdpf	Killing	Stopping container payload-processing
19:57:18	openshift-ingress	kubelet	e2e-ait-6e159107-openshift-default-6ccb5f479b-gc4q5	Killing	Stopping container istio-proxy
19:57:19	openshift-ingress	gateway_labeler_controller	e2e-derive-ee5db6ec	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-derive-ee5db6ec
19:57:19	openshift-ingress	deployment-controller	e2e-derive-ee5db6ec-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-derive-ee5db6ec-openshift-default-79ddf6f64d from 0 to 1
19:57:19	openshift-ingress	replicaset-controller	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d	SuccessfulCreate	Created pod: e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw
19:57:19	openshift-ingress	multus	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
19:57:22	openshift-ingress	deployment-controller	payload-processing-e2e-derive-ee5db6ec	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-derive-ee5db6ec-75689d8fbb from 0 to 1
19:57:22	openshift-ingress	deployment-controller	payload-pre-processing-e2e-derive-ee5db6ec	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d from 0 to 1
19:57:22 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-derive-ee5db6ec-75689d8fbb	FailedCreate	Error creating: pods "payload-processing-e2e-derive-ee5db6ec-75689d8fbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-ee5db6ec: serviceaccount "payload-processing-e2e-derive-ee5db6ec" not found
19:57:22	openshift-ingress	replicaset-controller	payload-processing-e2e-derive-ee5db6ec-75689d8fbb	SuccessfulCreate	Created pod: payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4
19:57:22 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d	FailedCreate	Error creating: pods "payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-ee5db6ec: serviceaccount "payload-processing-e2e-derive-ee5db6ec" not found
19:57:22	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d	SuccessfulCreate	Created pod: payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5
19:57:23	openshift-ingress	kubelet	payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	Created	Created container: payload-processing
19:57:23	openshift-ingress	kubelet	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	Created	Created container: payload-pre-processing
19:57:23	openshift-ingress	multus	payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
19:57:23	openshift-ingress	kubelet	payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:57:23	openshift-ingress	kubelet	payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	Started	Started container payload-processing
19:57:23	openshift-ingress	kubelet	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	Started	Started container payload-pre-processing
19:57:23	openshift-ingress	multus	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
19:57:23	openshift-ingress	kubelet	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:57:35 (x2)	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:57:35 (x2)	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Created	Created container: istio-proxy
19:57:35 (x2)	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Started	Started container istio-proxy
19:57:39 (x4)	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	ProbeError	Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused body:
19:57:39 (x4)	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Unhealthy	Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused
19:57:57	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-derive-ee5db6ec	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-derive-ee5db6ec: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-derive-ee5db6ec": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-derive-ee5db6ec, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 396e7b71-c905-4c1a-bac7-1c02ef093d0a, UID in object meta:
19:57:57	openshift-ingress	kubelet	e2e-derive-ee5db6ec-openshift-default-79ddf6f64d-zkpkw	Killing	Stopping container istio-proxy
19:57:57	openshift-ingress	kubelet	payload-processing-e2e-derive-ee5db6ec-75689d8fbb-qnqc4	Killing	Stopping container payload-processing
19:57:57	openshift-ingress	kubelet	payload-pre-processing-e2e-derive-ee5db6ec-5d9487884d-b4tb5	Killing	Stopping container payload-pre-processing
19:59:35	openshift-ingress	replicaset-controller	e2e-mt-c7847b84-openshift-default-69479fd9d8	SuccessfulCreate	Created pod: e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw
19:59:35	openshift-ingress	deployment-controller	e2e-mt-c7847b84-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-mt-c7847b84-openshift-default-69479fd9d8 from 0 to 1
19:59:35	openshift-ingress	GatewayAuthBootstrap	e2e-mt-c7847b84	ReconcileError	Failed to reconcile EnvoyFilter: failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-mt-c7847b84-authn-ssl: envoyfilters.networking.istio.io "e2e-mt-c7847b84-authn-ssl" already exists
19:59:35	openshift-ingress	gateway_labeler_controller	e2e-mt-c7847b84	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-mt-c7847b84
19:59:36	openshift-ingress	multus	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:59:39	openshift-ingress	multus	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
19:59:39	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	Started	Started container payload-pre-processing
19:59:39	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	Created	Created container: payload-pre-processing
19:59:39	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:59:39 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb	FailedCreate	Error creating: pods "payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c7847b84: serviceaccount "payload-processing-e2e-mt-c7847b84" not found
19:59:39 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b	FailedCreate	Error creating: pods "payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c7847b84: serviceaccount "payload-processing-e2e-mt-c7847b84" not found
19:59:39	openshift-ingress	kubelet	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
19:59:39	openshift-ingress	multus	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
19:59:39	openshift-ingress	kubelet	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	Created	Created container: payload-processing
19:59:39	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b	SuccessfulCreate	Created pod: payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq
19:59:39	openshift-ingress	replicaset-controller	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb	SuccessfulCreate	Created pod: payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44
19:59:39	openshift-ingress	kubelet	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	Started	Started container payload-processing
19:59:39	openshift-ingress	deployment-controller	payload-pre-processing-e2e-mt-c7847b84	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b from 0 to 1
19:59:39	openshift-ingress	deployment-controller	payload-processing-e2e-mt-c7847b84	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-mt-c7847b84-6cd6dcf8bb from 0 to 1
19:59:52 (x2)	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Started	Started container istio-proxy
19:59:52 (x2)	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Created	Created container: istio-proxy
19:59:52 (x2)	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
19:59:52	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	ProbeError	Readiness probe error: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused body:
19:59:52	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Unhealthy	Readiness probe failed: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused
19:59:57 (x4)	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	ProbeError	Startup probe error: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused body:
19:59:57 (x4)	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Unhealthy	Startup probe failed: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused
20:00:00	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-c7847b84-7c6c58cc9b-jqjwq	Killing	Stopping container payload-pre-processing
20:00:00	openshift-ingress	kubelet	payload-processing-e2e-mt-c7847b84-6cd6dcf8bb-z2k44	Killing	Stopping container payload-processing
20:00:00	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-mt-c7847b84	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-mt-c7847b84: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-mt-c7847b84": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-mt-c7847b84, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0a0b7205-5941-4ce4-af41-eb256cb17301, UID in object meta:
20:00:07	openshift-ingress	kubelet	e2e-mt-c7847b84-openshift-default-69479fd9d8-jfjmw	Killing	Stopping container istio-proxy
20:00:27	openshift-ingress	gateway_labeler_controller	e2e-mt-73760660	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-mt-73760660
20:00:28	openshift-ingress	deployment-controller	e2e-mt-73760660-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-mt-73760660-openshift-default-6dc79f5684 from 0 to 1
20:00:28	openshift-ingress	replicaset-controller	e2e-mt-73760660-openshift-default-6dc79f5684	SuccessfulCreate	Created pod: e2e-mt-73760660-openshift-default-6dc79f5684-95xgl
20:00:28	openshift-ingress	multus	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:00:31 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-mt-73760660-67c49cc76	FailedCreate	Error creating: pods "payload-pre-processing-e2e-mt-73760660-67c49cc76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-73760660: serviceaccount "payload-processing-e2e-mt-73760660" not found
20:00:31	openshift-ingress	deployment-controller	payload-processing-e2e-mt-73760660	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-mt-73760660-8ddd65d58 from 0 to 1
20:00:31	openshift-ingress	deployment-controller	payload-pre-processing-e2e-mt-73760660	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-mt-73760660-67c49cc76 from 0 to 1
20:00:31	openshift-ingress	replicaset-controller	payload-processing-e2e-mt-73760660-8ddd65d58	SuccessfulCreate	Created pod: payload-processing-e2e-mt-73760660-8ddd65d58-h9p57
20:00:31 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-mt-73760660-8ddd65d58	FailedCreate	Error creating: pods "payload-processing-e2e-mt-73760660-8ddd65d58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-73760660: serviceaccount "payload-processing-e2e-mt-73760660" not found
20:00:31	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-mt-73760660-67c49cc76	SuccessfulCreate	Created pod: payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6
20:00:32	openshift-ingress	kubelet	payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:00:32	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:00:32	openshift-ingress	multus	payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
20:00:32	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	Created	Created container: payload-pre-processing
20:00:32	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	Started	Started container payload-pre-processing
20:00:32	openshift-ingress	multus	payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
20:00:32	openshift-ingress	kubelet	payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	Started	Started container payload-processing
20:00:32	openshift-ingress	kubelet	payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	Created	Created container: payload-processing
20:00:44 (x2)	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Created	Created container: istio-proxy
20:00:44 (x2)	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:00:44 (x2)	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Started	Started container istio-proxy
20:00:48 (x4)	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Unhealthy	Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused
20:00:48 (x4)	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	ProbeError	Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused body:
20:01:20	openshift-ingress	kubelet	payload-processing-e2e-mt-73760660-8ddd65d58-h9p57	Killing	Stopping container payload-processing
20:01:20	openshift-ingress	kubelet	payload-pre-processing-e2e-mt-73760660-67c49cc76-blhp6	Killing	Stopping container payload-pre-processing
20:02:26	openshift-ingress	kubelet	e2e-mt-73760660-openshift-default-6dc79f5684-95xgl	Killing	Stopping container istio-proxy
20:03:22	openshift-ingress	replicaset-controller	e2e-inf-a-79961d-openshift-default-5cc96b8ff8	SuccessfulCreate	Created pod: e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw
20:03:22	openshift-ingress	deployment-controller	e2e-inf-a-79961d-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-inf-a-79961d-openshift-default-5cc96b8ff8 from 0 to 1
20:03:22	openshift-ingress	multus	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
20:03:22	openshift-ingress	gateway_labeler_controller	e2e-inf-a-79961d	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-79961d
20:03:26	openshift-ingress	deployment-controller	payload-processing-e2e-inf-a-79961d	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-inf-a-79961d-79f89db66 from 0 to 1
20:03:26 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-a-79961d-79f89db66	FailedCreate	Error creating: pods "payload-processing-e2e-inf-a-79961d-79f89db66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-79961d: serviceaccount "payload-processing-e2e-inf-a-79961d" not found
20:03:26	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb	SuccessfulCreate	Created pod: payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84
20:03:26 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb	FailedCreate	Error creating: pods "payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-79961d: serviceaccount "payload-processing-e2e-inf-a-79961d" not found
20:03:26	openshift-ingress	multus	payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
20:03:26	openshift-ingress	deployment-controller	payload-pre-processing-e2e-inf-a-79961d	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb from 0 to 1
20:03:26	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:03:26	openshift-ingress	multus	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
20:03:26	openshift-ingress	kubelet	payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:03:26	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-a-79961d-79f89db66	SuccessfulCreate	Created pod: payload-processing-e2e-inf-a-79961d-79f89db66-88g6l
20:03:27	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	Created	Created container: payload-pre-processing
20:03:27	openshift-ingress	kubelet	payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	Started	Started container payload-processing
20:03:27	openshift-ingress	kubelet	payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	Created	Created container: payload-processing
20:03:27	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	Started	Started container payload-pre-processing
20:03:30	openshift-ingress	gateway_labeler_controller	e2e-inf-b-08e715	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-08e715
20:03:30	openshift-ingress	replicaset-controller	e2e-inf-b-08e715-openshift-default-b59cd9464	SuccessfulCreate	Created pod: e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9
20:03:30	openshift-ingress	deployment-controller	e2e-inf-b-08e715-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-inf-b-08e715-openshift-default-b59cd9464 from 0 to 1
20:03:31	openshift-ingress	multus	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
20:03:34	openshift-ingress	deployment-controller	payload-processing-e2e-inf-b-08e715	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-inf-b-08e715-85bd995d5b from 0 to 1
20:03:35 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-b-08e715-85bd995d5b	FailedCreate	Error creating: pods "payload-processing-e2e-inf-b-08e715-85bd995d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-08e715: serviceaccount "payload-processing-e2e-inf-b-08e715" not found
20:03:35 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-b-08e715-5c98c45688	FailedCreate	Error creating: pods "payload-pre-processing-e2e-inf-b-08e715-5c98c45688-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-08e715: serviceaccount "payload-processing-e2e-inf-b-08e715" not found
20:03:35	openshift-ingress	multus	payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
20:03:35	openshift-ingress	kubelet	payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:03:35	openshift-ingress	replicaset-controller	payload-processing-e2e-inf-b-08e715-85bd995d5b	SuccessfulCreate	Created pod: payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8
20:03:35	openshift-ingress	kubelet	payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	Created	Created container: payload-processing
20:03:35	openshift-ingress	kubelet	payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	Started	Started container payload-processing
20:03:35	openshift-ingress	multus	payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
20:03:35	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:03:35	openshift-ingress	deployment-controller	payload-pre-processing-e2e-inf-b-08e715	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-inf-b-08e715-5c98c45688 from 0 to 1
20:03:35	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-inf-b-08e715-5c98c45688	SuccessfulCreate	Created pod: payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8
20:03:35	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	Created	Created container: payload-pre-processing
20:03:35	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	Started	Started container payload-pre-processing
20:03:39	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	ProbeError	Readiness probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused body:
20:03:39 (x2)	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:03:39	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Unhealthy	Readiness probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused
20:03:40 (x2)	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Started	Started container istio-proxy
20:03:40 (x2)	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Created	Created container: istio-proxy
20:03:44 (x4)	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	ProbeError	Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused body:
20:03:44 (x4)	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Unhealthy	Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused
20:03:48 (x2)	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Started	Started container istio-proxy
20:03:48 (x2)	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Created	Created container: istio-proxy
20:03:48 (x2)	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:03:48	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	ProbeError	Readiness probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused body:
20:03:48	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Unhealthy	Readiness probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused
20:03:52 (x3)	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Unhealthy	Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused
20:03:52 (x3)	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	ProbeError	Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused body:
20:05:14	openshift-ingress	kubelet	payload-processing-e2e-inf-a-79961d-79f89db66-88g6l	Killing	Stopping container payload-processing
20:05:14	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-a-79961d-65d6cb99cb-l9k84	Killing	Stopping container payload-pre-processing
20:05:14	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-inf-a-79961d	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-inf-a-79961d: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-inf-a-79961d": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-inf-a-79961d, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 6f2c1236-5150-4da9-b3d4-6446cefd6663, UID in object meta:
20:05:21	openshift-ingress	kubelet	e2e-inf-a-79961d-openshift-default-5cc96b8ff8-qtqhw	Killing	Stopping container istio-proxy
20:05:32	openshift-ingress	kubelet	payload-pre-processing-e2e-inf-b-08e715-5c98c45688-4tst8	Killing	Stopping container payload-pre-processing
20:05:32	openshift-ingress	kubelet	payload-processing-e2e-inf-b-08e715-85bd995d5b-j8fc8	Killing	Stopping container payload-processing
20:05:39	openshift-ingress	kubelet	e2e-inf-b-08e715-openshift-default-b59cd9464-9xqn9	Killing	Stopping container istio-proxy
20:05:40	openshift-ingress	multus	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:05:40	openshift-ingress	gateway_labeler_controller	e2e-api-a-3b36df	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-3b36df
20:05:40	openshift-ingress	deployment-controller	e2e-api-a-3b36df-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-api-a-3b36df-openshift-default-5b5fdcdb5b from 0 to 1
20:05:40	openshift-ingress	replicaset-controller	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b	SuccessfulCreate	Created pod: e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	Created	Created container: payload-pre-processing
20:05:44	openshift-ingress	multus	payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	AddedInterface	Add eth0 [10.133.0.66/23] from ovn-kubernetes
20:05:44	openshift-ingress	deployment-controller	payload-pre-processing-e2e-api-a-3b36df	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-api-a-3b36df-585f45c88d from 0 to 1
20:05:44	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-api-a-3b36df-585f45c88d	SuccessfulCreate	Created pod: payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb
20:05:44 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-api-a-3b36df-585f45c88d	FailedCreate	Error creating: pods "payload-pre-processing-e2e-api-a-3b36df-585f45c88d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3b36df: serviceaccount "payload-processing-e2e-api-a-3b36df" not found
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	Started	Started container payload-pre-processing
20:05:44	openshift-ingress	deployment-controller	payload-processing-e2e-api-a-3b36df	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-api-a-3b36df-6bf557cb44 from 0 to 1
20:05:44	openshift-ingress	replicaset-controller	payload-processing-e2e-api-a-3b36df-6bf557cb44	SuccessfulCreate	Created pod: payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g
20:05:44	openshift-ingress	kubelet	payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	Started	Started container payload-processing
20:05:44	openshift-ingress	kubelet	payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	Created	Created container: payload-processing
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:05:44	openshift-ingress	multus	payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	AddedInterface	Add eth0 [10.133.0.67/23] from ovn-kubernetes
20:05:44 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-api-a-3b36df-6bf557cb44	FailedCreate	Error creating: pods "payload-processing-e2e-api-a-3b36df-6bf557cb44-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3b36df: serviceaccount "payload-processing-e2e-api-a-3b36df" not found
20:05:44	openshift-ingress	kubelet	payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:05:48	openshift-ingress	gateway_labeler_controller	e2e-api-b-d978f1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-d978f1
20:05:48	openshift-ingress	replicaset-controller	e2e-api-b-d978f1-openshift-default-9b65d484	SuccessfulCreate	Created pod: e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg
20:05:48	openshift-ingress	deployment-controller	e2e-api-b-d978f1-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-api-b-d978f1-openshift-default-9b65d484 from 0 to 1
20:05:49	openshift-ingress	multus	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
20:05:52	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48	SuccessfulCreate	Created pod: payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss
20:05:52	openshift-ingress	deployment-controller	payload-pre-processing-e2e-api-b-d978f1	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48 from 0 to 1
20:05:52 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48	FailedCreate	Error creating: pods "payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-d978f1: serviceaccount "payload-processing-e2e-api-b-d978f1" not found
20:05:52 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-api-b-d978f1-658d59987b	FailedCreate	Error creating: pods "payload-processing-e2e-api-b-d978f1-658d59987b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-d978f1: serviceaccount "payload-processing-e2e-api-b-d978f1" not found
20:05:52	openshift-ingress	deployment-controller	payload-processing-e2e-api-b-d978f1	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-api-b-d978f1-658d59987b from 0 to 1
20:05:52	openshift-ingress	replicaset-controller	payload-processing-e2e-api-b-d978f1-658d59987b	SuccessfulCreate	Created pod: payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q
20:05:53	openshift-ingress	multus	payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	AddedInterface	Add eth0 [10.133.0.69/23] from ovn-kubernetes
20:05:53	openshift-ingress	kubelet	payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:05:53	openshift-ingress	kubelet	payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	Created	Created container: payload-processing
20:05:53	openshift-ingress	kubelet	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	Started	Started container payload-pre-processing
20:05:53	openshift-ingress	kubelet	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	Created	Created container: payload-pre-processing
20:05:53	openshift-ingress	kubelet	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:05:53	openshift-ingress	multus	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	AddedInterface	Add eth0 [10.133.0.68/23] from ovn-kubernetes
20:05:53	openshift-ingress	kubelet	payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	Started	Started container payload-processing
20:05:58	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	ProbeError	Readiness probe error: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:05:58	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Unhealthy	Readiness probe failed: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:06:00 (x2)	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:06:00 (x2)	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Started	Started container istio-proxy
20:06:00 (x2)	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Created	Created container: istio-proxy
20:06:05 (x4)	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Unhealthy	Startup probe failed: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused
20:06:05 (x4)	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	ProbeError	Startup probe error: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused body:
20:06:07	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:06:07	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	ProbeError	Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:06:08 (x2)	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:06:08 (x2)	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Created	Created container: istio-proxy
20:06:08 (x2)	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Started	Started container istio-proxy
20:06:11 (x3)	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	ProbeError	Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
20:06:11 (x3)	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Unhealthy	Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused
20:06:20	openshift-ingress	kubelet	payload-processing-e2e-api-a-3b36df-6bf557cb44-2n52g	Killing	Stopping container payload-processing
20:06:20	openshift-ingress	kubelet	payload-pre-processing-e2e-api-a-3b36df-585f45c88d-fzttb	Killing	Stopping container payload-pre-processing
20:06:26	openshift-ingress	kubelet	e2e-api-a-3b36df-openshift-default-5b5fdcdb5b-47xxw	Killing	Stopping container istio-proxy
20:06:42	openshift-ingress	kubelet	payload-pre-processing-e2e-api-b-d978f1-54fb5b9f48-lmrss	Killing	Stopping container payload-pre-processing
20:06:42	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-api-b-d978f1	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-api-b-d978f1: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-api-b-d978f1": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-api-b-d978f1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d251b13c-0d9e-4053-bc04-e14df819ad93, UID in object meta:
20:06:42	openshift-ingress	kubelet	payload-processing-e2e-api-b-d978f1-658d59987b-wwm6q	Killing	Stopping container payload-processing
20:06:49	openshift-ingress	kubelet	e2e-api-b-d978f1-openshift-default-9b65d484-j4bgg	Killing	Stopping container istio-proxy
20:06:50	openshift-ingress	deployment-controller	e2e-shared-a-d3c546-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-d3c546-openshift-default-7989484756 from 0 to 1
20:06:50	openshift-ingress	replicaset-controller	e2e-shared-a-d3c546-openshift-default-7989484756	SuccessfulCreate	Created pod: e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5
20:06:50	openshift-ingress	multus	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
20:06:50	openshift-ingress	gateway_labeler_controller	e2e-shared-a-d3c546	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-d3c546
20:06:54	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-d3c546	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-d3c546-68dbffb748 from 0 to 1
20:06:54	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw
20:06:54	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-d3c546	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-d3c546-6b6f876dfd from 0 to 1
20:06:54	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-d3c546-6b6f876dfd	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d
20:06:54 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-d3c546-6b6f876dfd	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-d3c546-6b6f876dfd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-d3c546: serviceaccount "payload-processing-e2e-shared-a-d3c546" not found
20:06:54 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-d3c546: serviceaccount "payload-processing-e2e-shared-a-d3c546" not found
20:06:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:06:55	openshift-ingress	multus	payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	AddedInterface	Add eth0 [10.133.0.71/23] from ovn-kubernetes
20:06:55	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	AddedInterface	Add eth0 [10.133.0.70/23] from ovn-kubernetes
20:06:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	Created	Created container: payload-pre-processing
20:06:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	Started	Started container payload-pre-processing
20:06:55	openshift-ingress	kubelet	payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	Started	Started container payload-processing
20:06:55	openshift-ingress	kubelet	payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	Created	Created container: payload-processing
20:06:55	openshift-ingress	kubelet	payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:06:58	openshift-ingress	gateway_labeler_controller	e2e-shared-b-73ce16	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-73ce16
20:06:58	openshift-ingress	deployment-controller	e2e-shared-b-73ce16-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-73ce16-openshift-default-56c85b55b from 0 to 1
20:06:58	openshift-ingress	replicaset-controller	e2e-shared-b-73ce16-openshift-default-56c85b55b	SuccessfulCreate	Created pod: e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg
20:06:59	openshift-ingress	multus	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
20:07:02 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-73ce16-975d66488	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-73ce16-975d66488-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-73ce16: serviceaccount "payload-processing-e2e-shared-b-73ce16" not found
20:07:02	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-73ce16	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-73ce16-975d66488 from 0 to 1
20:07:02	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-73ce16	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-73ce16-5979bbc9bb from 0 to 1
20:07:02 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-73ce16-5979bbc9bb	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-73ce16-5979bbc9bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-73ce16: serviceaccount "payload-processing-e2e-shared-b-73ce16" not found
20:07:03	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:07:03	openshift-ingress	multus	payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	AddedInterface	Add eth0 [10.133.0.73/23] from ovn-kubernetes
20:07:03	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	Started	Started container payload-pre-processing
20:07:03	openshift-ingress	kubelet	payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	Created	Created container: payload-processing
20:07:03	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	Created	Created container: payload-pre-processing
20:07:03	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-73ce16-5979bbc9bb	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb
20:07:03	openshift-ingress	kubelet	payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:07:03	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	AddedInterface	Add eth0 [10.133.0.72/23] from ovn-kubernetes
20:07:03	openshift-ingress	kubelet	payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	Started	Started container payload-processing
20:07:03	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-73ce16-975d66488	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j
20:07:07 (x2)	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Started	Started container istio-proxy
20:07:07 (x2)	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Created	Created container: istio-proxy
20:07:07 (x2)	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:07:11 (x4)	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Unhealthy	Startup probe failed: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused
20:07:11 (x4)	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	ProbeError	Startup probe error: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused body:
20:07:17	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	ProbeError	Readiness probe error: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:07:17	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Unhealthy	Readiness probe failed: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:07:18 (x2)	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Started	Started container istio-proxy
20:07:18 (x2)	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:07:18 (x2)	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Created	Created container: istio-proxy
20:07:23 (x4)	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Unhealthy	Startup probe failed: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused
20:07:23 (x4)	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	ProbeError	Startup probe error: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused body:
20:16:17	openshift-ingress	multus	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
20:16:17	openshift-ingress	replicaset-controller	e2e-ipp-a-018406-openshift-default-57cd545f48	SuccessfulCreate	Created pod: e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh
20:16:17	openshift-ingress	deployment-controller	e2e-ipp-a-018406-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-ipp-a-018406-openshift-default-57cd545f48 from 0 to 1
20:16:17	openshift-ingress	gateway_labeler_controller	e2e-ipp-a-018406	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-018406
20:16:20	openshift-ingress	deployment-controller	payload-pre-processing-e2e-ipp-a-018406	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8 from 0 to 1
20:16:20	openshift-ingress	deployment-controller	payload-processing-e2e-ipp-a-018406	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-ipp-a-018406-75754c7976 from 0 to 1
20:16:21	openshift-ingress	kubelet	payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:16:21	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8	SuccessfulCreate	Created pod: payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h
20:16:21	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-a-018406-75754c7976	SuccessfulCreate	Created pod: payload-processing-e2e-ipp-a-018406-75754c7976-84hjb
20:16:21 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8	FailedCreate	Error creating: pods "payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-018406: serviceaccount "payload-processing-e2e-ipp-a-018406" not found
20:16:21	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:16:21 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-a-018406-75754c7976	FailedCreate	Error creating: pods "payload-processing-e2e-ipp-a-018406-75754c7976-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-018406: serviceaccount "payload-processing-e2e-ipp-a-018406" not found
20:16:21	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	Started	Started container payload-pre-processing
20:16:21	openshift-ingress	multus	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	AddedInterface	Add eth0 [10.133.0.84/23] from ovn-kubernetes
20:16:21	openshift-ingress	multus	payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	AddedInterface	Add eth0 [10.133.0.85/23] from ovn-kubernetes
20:16:21	openshift-ingress	kubelet	payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	Created	Created container: payload-processing
20:16:21	openshift-ingress	kubelet	payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	Started	Started container payload-processing
20:16:21	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	Created	Created container: payload-pre-processing
20:16:35	openshift-ingress	replicaset-controller	e2e-ipp-b-933254-openshift-default-5487f77964	SuccessfulCreate	Created pod: e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5
20:16:35	openshift-ingress	deployment-controller	e2e-ipp-b-933254-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-ipp-b-933254-openshift-default-5487f77964 from 0 to 1
20:16:35	openshift-ingress	gateway_labeler_controller	e2e-ipp-b-933254	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-933254
20:16:36	openshift-ingress	multus	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
20:16:38 (x2)	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Created	Created container: istio-proxy
20:16:38 (x2)	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:16:38 (x2)	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Started	Started container istio-proxy
20:16:39	openshift-ingress	multus	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	AddedInterface	Add eth0 [10.133.0.86/23] from ovn-kubernetes
20:16:39	openshift-ingress	deployment-controller	payload-processing-e2e-ipp-b-933254	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-ipp-b-933254-c4b5858d4 from 0 to 1
20:16:39	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-b-933254-c4b5858d4	SuccessfulCreate	Created pod: payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2
20:16:39 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4	FailedCreate	Error creating: pods "payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-933254: serviceaccount "payload-processing-e2e-ipp-b-933254" not found
20:16:39	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4	SuccessfulCreate	Created pod: payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259
20:16:39	openshift-ingress	deployment-controller	payload-pre-processing-e2e-ipp-b-933254	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-ipp-b-933254-78765ccbd4 from 0 to 1
20:16:39 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-b-933254-c4b5858d4	FailedCreate	Error creating: pods "payload-processing-e2e-ipp-b-933254-c4b5858d4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-933254: serviceaccount "payload-processing-e2e-ipp-b-933254" not found
20:16:40	openshift-ingress	multus	payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	AddedInterface	Add eth0 [10.133.0.87/23] from ovn-kubernetes
20:16:40	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	Started	Started container payload-pre-processing
20:16:40	openshift-ingress	kubelet	payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	Started	Started container payload-processing
20:16:40	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	Created	Created container: payload-pre-processing
20:16:40	openshift-ingress	kubelet	payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	Created	Created container: payload-processing
20:16:40	openshift-ingress	kubelet	payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:16:40	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:16:42 (x4)	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	ProbeError	Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused body:
20:16:42 (x4)	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Unhealthy	Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused
20:16:53	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Unhealthy	Readiness probe failed: Get "http://10.134.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:16:53	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	ProbeError	Readiness probe error: Get "http://10.134.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:16:54 (x2)	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Started	Started container istio-proxy
20:16:54 (x2)	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Created	Created container: istio-proxy
20:16:54 (x2)	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:16:59 (x4)	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Unhealthy	Startup probe failed: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused
20:16:59 (x4)	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	ProbeError	Startup probe error: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused body:
20:17:50	openshift-ingress	replicaset-controller	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b	SuccessfulCreate	Created pod: e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m
20:17:50	openshift-ingress	deployment-controller	e2e-ipp-cleanup-394c6a-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b from 0 to 1
20:17:50	openshift-ingress	gateway_labeler_controller	e2e-ipp-cleanup-394c6a	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-cleanup-394c6a
20:17:51	openshift-ingress	multus	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	AddedInterface	Add eth0 [10.132.0.62/23] from ovn-kubernetes
20:17:54	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76	SuccessfulCreate	Created pod: payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6
20:17:54 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76	FailedCreate	Error creating: pods "payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-394c6a: serviceaccount "payload-processing-e2e-ipp-cleanup-394c6a" not found
20:17:54	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd	SuccessfulCreate	Created pod: payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj
20:17:54 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd	FailedCreate	Error creating: pods "payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-394c6a: serviceaccount "payload-processing-e2e-ipp-cleanup-394c6a" not found
20:17:54	openshift-ingress	deployment-controller	payload-pre-processing-e2e-ipp-cleanup-394c6a	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd from 0 to 1
20:17:54	openshift-ingress	deployment-controller	payload-processing-e2e-ipp-cleanup-394c6a	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-ipp-cleanup-394c6a-548646bc76 from 0 to 1
20:17:55	openshift-ingress	multus	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	AddedInterface	Add eth0 [10.133.0.89/23] from ovn-kubernetes
20:17:55	openshift-ingress	kubelet	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:17:55	openshift-ingress	kubelet	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	Created	Created container: payload-processing
20:17:55	openshift-ingress	kubelet	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	Started	Started container payload-processing
20:17:55	openshift-ingress	multus	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	AddedInterface	Add eth0 [10.133.0.90/23] from ovn-kubernetes
20:17:55	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	Started	Started container payload-pre-processing
20:17:55	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	Pulled	Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
20:17:55	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	Created	Created container: payload-pre-processing
20:18:06 (x2)	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Created	Created container: istio-proxy
20:18:06 (x2)	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
20:18:06 (x2)	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Started	Started container istio-proxy
20:18:11 (x4)	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	ProbeError	Startup probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused body:
20:18:11 (x4)	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Unhealthy	Startup probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused
20:18:19	openshift-ingress	kubelet	payload-processing-e2e-ipp-cleanup-394c6a-548646bc76-vpwf6	Killing	Stopping container payload-processing
20:18:19	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-cleanup-394c6a-fdb5656bd-pkhdj	Killing	Stopping container payload-pre-processing
20:18:26	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Killing	Stopping container istio-proxy
20:18:27	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:18:27	openshift-ingress	kubelet	e2e-ipp-cleanup-394c6a-openshift-default-7cf5b4f69b-5529m	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:18:45	openshift-ingress	kubelet	payload-processing-e2e-ipp-a-018406-75754c7976-84hjb	Killing	Stopping container payload-processing
20:18:45	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-a-018406-5fc4cbcbd8-5z95h	Killing	Stopping container payload-pre-processing
20:19:40	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Killing	Stopping container istio-proxy
20:19:43	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:19:43	openshift-ingress	kubelet	e2e-ipp-a-018406-openshift-default-57cd545f48-6t5wh	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:19:51	openshift-ingress	kubelet	payload-pre-processing-e2e-ipp-b-933254-78765ccbd4-m6259	Killing	Stopping container payload-pre-processing
20:19:51	openshift-ingress	kubelet	payload-processing-e2e-ipp-b-933254-c4b5858d4-8qtk2	Killing	Stopping container payload-processing
20:19:57	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Killing	Stopping container istio-proxy
20:20:00	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:20:00	openshift-ingress	kubelet	e2e-ipp-b-933254-openshift-default-5487f77964-q4dh5	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:20:08	openshift-ingress	kubelet	payload-processing-e2e-shared-a-d3c546-6b6f876dfd-tg88d	Killing	Stopping container payload-processing
20:20:08	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-d3c546-68dbffb748-g5tfw	Killing	Stopping container payload-pre-processing
20:20:08	openshift-ingress	endpoint-controller	payload-pre-processing-e2e-shared-a-d3c546	FailedToUpdateEndpoint	Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-shared-a-d3c546: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-shared-a-d3c546": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-shared-a-d3c546, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d4d90beb-325c-45d3-8c8d-146caf181886, UID in object meta:
20:20:15	openshift-ingress	kubelet	e2e-shared-a-d3c546-openshift-default-7989484756-cmjn5	Killing	Stopping container istio-proxy
20:20:26	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-73ce16-975d66488-vs24j	Killing	Stopping container payload-pre-processing
20:20:26	openshift-ingress	kubelet	payload-processing-e2e-shared-b-73ce16-5979bbc9bb-lcrpb	Killing	Stopping container payload-processing
20:20:33	openshift-ingress	kubelet	e2e-shared-b-73ce16-openshift-default-56c85b55b-kdlqg	Killing	Stopping container istio-proxy