Events

Time	Namespace	Component	RelatedObject	Reason	Message
11:59:07	openshift-ingress		istiod-openshift-gateway-7876bfcb74-h94gb	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-h94gb to ip-10-0-143-70.ec2.internal
12:08:51	openshift-ingress		e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c to ip-10-0-143-70.ec2.internal
11:59:14	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-fwvvf	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fwvvf to ip-10-0-131-114.ec2.internal
12:05:46	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-gnw8g	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-gnw8g to ip-10-0-143-70.ec2.internal
12:06:46	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-j8gnl	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-j8gnl to ip-10-0-131-114.ec2.internal
12:05:46	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-qmz72	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-qmz72 to ip-10-0-128-8.ec2.internal
12:06:01	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-rscbr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-rscbr to ip-10-0-128-8.ec2.internal
12:06:01	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-thwmn	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-thwmn to ip-10-0-143-70.ec2.internal
12:06:46	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-w85rr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w85rr to ip-10-0-128-8.ec2.internal
12:08:31	openshift-ingress		e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n to ip-10-0-143-70.ec2.internal
12:01:39	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg4xk to ip-10-0-131-114.ec2.internal
12:06:16	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-44fbq	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-44fbq to ip-10-0-131-114.ec2.internal
12:00:40	openshift-ingress		data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6 to ip-10-0-131-114.ec2.internal
11:49:01	openshift-ingress		router-default-7bf56d884f-q94vb	FailedScheduling	no nodes available to schedule pods
12:00:40	openshift-ingress		kube-auth-proxy-74f544c66f-xq9bc	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-74f544c66f-xq9bc to ip-10-0-128-8.ec2.internal
12:06:01	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-566g7	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-566g7 to ip-10-0-131-114.ec2.internal
11:59:14	openshift-ingress		openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw to ip-10-0-131-114.ec2.internal
12:04:54	openshift-ingress		payload-pre-processing-7b98bb678c-xwtgm	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-xwtgm to ip-10-0-143-70.ec2.internal
12:06:16	openshift-ingress		maas-default-gateway-openshift-default-7854949c9-8ncd4	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8ncd4 to ip-10-0-143-70.ec2.internal
12:08:37	openshift-ingress		payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7 to ip-10-0-143-70.ec2.internal
12:08:57	openshift-ingress		payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d to ip-10-0-143-70.ec2.internal
12:04:54	openshift-ingress		payload-processing-5f78687d5b-2z78c	Scheduled	Successfully assigned openshift-ingress/payload-processing-5f78687d5b-2z78c to ip-10-0-143-70.ec2.internal
12:08:37	openshift-ingress		payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v to ip-10-0-143-70.ec2.internal
12:08:57	openshift-ingress		payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7 to ip-10-0-128-8.ec2.internal
12:00:40	openshift-ingress		kube-auth-proxy-74f544c66f-zbpk8	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-74f544c66f-zbpk8 to ip-10-0-131-114.ec2.internal
11:48:59	openshift-ingress		router-default-7bf56d884f-q94vb	FailedScheduling	no nodes available to schedule pods
11:55:47	openshift-ingress		router-default-7bf56d884f-q94vb	Scheduled	Successfully assigned openshift-ingress/router-default-7bf56d884f-q94vb to ip-10-0-143-70.ec2.internal
11:48:26	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
11:48:26	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
11:48:26	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-7bf56d884f from 0 to 1
11:48:31	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
11:48:36	openshift-ingress	replicaset-controller	router-default-7bf56d884f	FailedCreate	Error creating: pods "router-default-7bf56d884f-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range
11:48:59	openshift-ingress	replicaset-controller	router-default-7bf56d884f	SuccessfulCreate	Created pod: router-default-7bf56d884f-q94vb
11:55:27 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
11:55:55 (x5)	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
11:55:55 (x5)	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
11:55:59	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
11:56:01 (x3)	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
11:56:04	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
11:56:04	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	Started	Container started
11:56:04	openshift-ingress	kubelet	router-default-7bf56d884f-q94vb	Created	Container created
11:56:04	openshift-ingress	multus	router-default-7bf56d884f-q94vb	AddedInterface	Add eth0 [10.132.0.19/23] from ovn-kubernetes
11:59:06	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
11:59:07	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-h94gb
11:59:07	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
11:59:08	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-h94gb	AddedInterface	Add eth0 [10.132.0.25/23] from ovn-kubernetes
11:59:08	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
11:59:13	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.951s (4.951s including waiting). Image size: 185476477 bytes.
11:59:13	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Created	Container created
11:59:13	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Started	Container started
11:59:14	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-6f8685f4cf	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw
11:59:14	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-fwvvf
11:59:14	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
11:59:14	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-6f8685f4cf from 0 to 1
11:59:14	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
11:59:14	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7854949c9 from 0 to 1
11:59:14	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:59:15	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
11:59:15	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-fwvvf	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
11:59:15	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
11:59:15	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
11:59:17	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
11:59:25	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	Created	Container created
11:59:25	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.166s (10.166s including waiting). Image size: 183555333 bytes.
11:59:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.285s (10.285s including waiting). Image size: 183555333 bytes.
11:59:25	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6f8685f4cf-8nkpw	Started	Container started
11:59:30	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:59:30	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:59:45	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:59:45	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:00:40	openshift-ingress	replicaset-controller	kube-auth-proxy-74f544c66f	SuccessfulCreate	Created pod: kube-auth-proxy-74f544c66f-xq9bc
12:00:40	openshift-ingress	replicaset-controller	kube-auth-proxy-74f544c66f	SuccessfulCreate	Created pod: kube-auth-proxy-74f544c66f-zbpk8
12:00:40	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-7d45d5c789 from 0 to 1
12:00:40	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-7d45d5c789	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6
12:00:40	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-74f544c66f from 0 to 2
12:00:41	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-zbpk8	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
12:00:41	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6	Started	Container started
12:00:41	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
12:00:41	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:00:41	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d45d5c789s5bx6	Created	Container created
12:00:41	openshift-ingress	multus	kube-auth-proxy-74f544c66f-zbpk8	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
12:00:41	openshift-ingress	multus	kube-auth-proxy-74f544c66f-xq9bc	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
12:00:41	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-xq9bc	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
12:00:43	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-xq9bc	Started	Container started
12:00:43	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-xq9bc	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.211s (2.211s including waiting). Image size: 188904677 bytes.
12:00:43	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-xq9bc	Created	Container created
12:00:44	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-zbpk8	Started	Container started
12:00:44	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-zbpk8	Created	Container created
12:00:44	openshift-ingress	kubelet	kube-auth-proxy-74f544c66f-zbpk8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.884s (2.884s including waiting). Image size: 188904677 bytes.
12:00:55	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:00:55	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:10	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:01:10	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:01:27 (x3)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:27 (x3)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:39	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
12:01:39	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-jg4xk
12:01:40	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
12:01:40	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
12:01:43	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Started	Container started
12:01:43	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.277s (3.277s including waiting). Image size: 185476477 bytes.
12:01:43	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Created	Container created
12:01:44	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:01:44	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-jg4xk	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
12:01:45	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-h94gb	Killing	Stopping container discovery
12:01:45	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-h94gb
12:01:45	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
12:01:53 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:53 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:08	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:08	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:52 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Created	Container created
12:02:52 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Started	Container started
12:02:52	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:02:58 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	ProbeError	Startup probe error: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused body:
12:02:58 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Unhealthy	Startup probe failed: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused
12:03:00 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:00 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:15 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:15 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:04:54	openshift-ingress	multus	payload-pre-processing-7b98bb678c-xwtgm	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
12:04:54	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-xwtgm
12:04:54	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5f78687d5b from 0 to 1
12:04:54	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	SuccessfulCreate	Created pod: payload-processing-5f78687d5b-2z78c
12:04:54 (x6)	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	FailedCreate	Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
12:04:54	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-xwtgm	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
12:04:54	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
12:04:54 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
12:04:55	openshift-ingress	multus	payload-processing-5f78687d5b-2z78c	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
12:04:55	openshift-ingress	kubelet	payload-processing-5f78687d5b-2z78c	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
12:04:57	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-xwtgm	Started	Container started
12:04:57	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-xwtgm	Created	Container created
12:04:57	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-xwtgm	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.276s (2.276s including waiting). Image size: 125317101 bytes.
12:04:57	openshift-ingress	kubelet	payload-processing-5f78687d5b-2z78c	Started	Container started
12:04:57	openshift-ingress	kubelet	payload-processing-5f78687d5b-2z78c	Created	Container created
12:04:57	openshift-ingress	kubelet	payload-processing-5f78687d5b-2z78c	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.15s (2.15s including waiting). Image size: 125317101 bytes.
12:05:46	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-gnw8g	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
12:05:46	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7854949c9 from 1 to 3
12:05:46	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 3; reason: cpu resource utilization (percentage of request) above target
12:05:46	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-gnw8g
12:05:46	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-qmz72
12:05:46	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
12:05:46	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
12:05:46	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-qmz72	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Created	Container created
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.714s (2.714s including waiting). Image size: 183555333 bytes.
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Created	Container created
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Started	Container started
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Started	Container started
12:05:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.65s (2.65s including waiting). Image size: 183555333 bytes.
12:05:56 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	ProbeError	Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused body:
12:05:56 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Unhealthy	Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused
12:05:56 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Unhealthy	Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused
12:05:56 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	ProbeError	Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused body:
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	Created	Container created
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	Started	Container started
12:06:01	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-566g7	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:01	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7854949c9 from 3 to 6
12:06:01	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 6; reason: cpu resource utilization (percentage of request) above target
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	Created	Container created
12:06:01	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-thwmn	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	Started	Container started
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	Created	Container created
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	Started	Container started
12:06:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-rscbr
12:06:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-566g7
12:06:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-thwmn
12:06:01	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-rscbr	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
12:06:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:08 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	Unhealthy	Startup probe failed: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused
12:06:08 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	Unhealthy	Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused
12:06:08 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	ProbeError	Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused body:
12:06:08 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	ProbeError	Startup probe error: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused body:
12:06:08 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	ProbeError	Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused body:
12:06:08 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	Unhealthy	Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	Created	Container created
12:06:16	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-44fbq
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	Created	Container created
12:06:16	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7854949c9 from 6 to 8
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:16	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-44fbq	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
12:06:16	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-8ncd4
12:06:16	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-8ncd4	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
12:06:16	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason:
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	Started	Container started
12:06:16	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	Started	Container started
12:06:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	Unhealthy	Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused
12:06:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	ProbeError	Startup probe error: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused body:
12:06:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	Unhealthy	Startup probe failed: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused
12:06:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	ProbeError	Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused body:
12:06:41	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	Unhealthy	Readiness probe failed: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:06:41	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-thwmn	ProbeError	Readiness probe error: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:06:46	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-j8gnl	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
12:06:46	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-7854949c9-j8gnl
12:06:46	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7854949c9	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7854949c9-w85rr
12:06:46	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
12:06:46	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7854949c9 from 8 to 10
12:06:46	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-w85rr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:46	openshift-ingress	multus	maas-default-gateway-openshift-default-7854949c9-w85rr	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
12:06:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-j8gnl	Created	Container created
12:06:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-w85rr	Started	Container started
12:06:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-w85rr	Created	Container created
12:06:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-j8gnl	Started	Container started
12:06:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-j8gnl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:06:54 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-w85rr	Unhealthy	Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused
12:06:54 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-w85rr	ProbeError	Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused body:
12:06:54 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-j8gnl	ProbeError	Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused body:
12:06:54 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-j8gnl	Unhealthy	Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused
12:08:31	openshift-ingress	multus	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
12:08:31	openshift-ingress	replicaset-controller	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947	SuccessfulCreate	Created pod: e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n
12:08:31	openshift-ingress	deployment-controller	e2e-shared-a-main-8d0914-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-8d0914-openshift-default-6bb45fd947 from 0 to 1
12:08:31	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-8d0914	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-8d0914
12:08:36	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-8d0914	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-8d0914-785c597596 from 0 to 1
12:08:36	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-8d0914	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc9 from 0 to 1
12:08:37	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
12:08:37	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	Started	Container started
12:08:37 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-8d0914-785c597596	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-8d0914-785c597596-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-8d0914: serviceaccount "payload-processing-e2e-shared-a-main-8d0914" not found
12:08:37	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-8d0914-785c597596	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v
12:08:37	openshift-ingress	multus	payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
12:08:37	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	Created	Container created
12:08:37 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc9	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-8d0914: serviceaccount "payload-processing-e2e-shared-a-main-8d0914" not found
12:08:37	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc9	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7
12:08:37	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	Started	Container started
12:08:37	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
12:08:37	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
12:08:37	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	Created	Container created
12:08:50 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Created	Container created
12:08:50	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	ProbeError	Readiness probe error: Get "http://10.132.0.53:15021/healthz/ready": read tcp 10.132.0.2:33244->10.132.0.53:15021: read: connection reset by peer body:
12:08:50	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Unhealthy	Readiness probe failed: Get "http://10.132.0.53:15021/healthz/ready": read tcp 10.132.0.2:33244->10.132.0.53:15021: read: connection reset by peer
12:08:50 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Started	Container started
12:08:50 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:08:51	openshift-ingress	deployment-controller	e2e-shared-b-main-4dc002-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-4dc002-openshift-default-d98b4775 from 0 to 1
12:08:51	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-4dc002	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-4dc002
12:08:51	openshift-ingress	replicaset-controller	e2e-shared-b-main-4dc002-openshift-default-d98b4775	SuccessfulCreate	Created pod: e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c
12:08:51	openshift-ingress	multus	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
12:08:56	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-4dc002	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-4dc002-57855bfc7b from 0 to 1
12:08:56 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	ProbeError	Startup probe error: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused body:
12:08:56 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Unhealthy	Startup probe failed: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused
12:08:56	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-4dc002	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568c from 0 to 1
12:08:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
12:08:57	openshift-ingress	multus	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
12:08:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Started	Container started
12:08:57	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7
12:08:57	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
12:08:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Created	Container created
12:08:57 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-4dc002: serviceaccount "payload-processing-e2e-shared-b-main-4dc002" not found
12:08:57 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568c	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-4dc002: serviceaccount "payload-processing-e2e-shared-b-main-4dc002" not found
12:08:57	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568c	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d
12:08:57	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
12:08:59	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.034s (2.034s including waiting). Image size: 125317101 bytes.
12:09:00	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Started	Container started
12:09:00	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Created	Container created
12:09:09	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	ProbeError	Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:09:09	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:09:12 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
12:09:12 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Created	Container created
12:09:13 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Started	Container started
12:09:16 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-8d0914-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:09:16 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-8d0914-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:09:18 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	ProbeError	Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
12:09:18 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Unhealthy	Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused
12:09:21 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-4dc002-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:09:21 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-4dc002-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:09:36	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-4dc002-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:09:36	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-4dc002-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:10:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	Unhealthy	Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	Unhealthy	Readiness probe failed: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	ProbeError	Readiness probe error: Get "http://10.132.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	ProbeError	Readiness probe error: Get "http://10.133.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-566g7	Unhealthy	Readiness probe failed: Get "http://10.133.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:10:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-rscbr	ProbeError	Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-44fbq	ProbeError	Readiness probe error: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:10:56 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-8ncd4	Unhealthy	Readiness probe failed: Get "http://10.132.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:12:43 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	ProbeError	Readiness probe error: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:12:43 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-qmz72	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:12:43 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	ProbeError	Readiness probe error: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:12:43 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-gnw8g	Unhealthy	Readiness probe failed: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:12:45 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	Unhealthy	Readiness probe failed: Get "http://10.133.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
12:12:45 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7854949c9-fwvvf	ProbeError	Readiness probe error: Get "http://10.133.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
12:12:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d0914-7b8c5bdbc99vvs7	Killing	Stopping container payload-pre-processing
12:12:55	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d0914-785c597596-cj59v	Killing	Stopping container payload-processing
12:13:05	openshift-ingress	kubelet	e2e-shared-a-main-8d0914-openshift-default-6bb45fd947-vh76n	Killing	Stopping container istio-proxy
12:13:15	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Killing	Stopping container payload-processing
12:13:16	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Killing	Stopping container payload-pre-processing
12:13:24	openshift-ingress	kubelet	e2e-shared-b-main-4dc002-openshift-default-d98b4775-x594c	Killing	Stopping container istio-proxy
12:13:31 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-4dc002-57855bfc7b-l52p7	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
12:13:40 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-4dc002-5ff4cf568cxjq7d	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")