Events

Time	Namespace	Component	RelatedObject	Reason	Message
23:39:41	openshift-ingress		istiod-openshift-gateway-7876bfcb74-48t7g	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-48t7g to ip-10-0-138-47.ec2.internal
23:41:12	openshift-ingress		kube-auth-proxy-798f575fff-9ktzc	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-798f575fff-9ktzc to ip-10-0-140-219.ec2.internal
23:46:32	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rrg7c to ip-10-0-138-142.ec2.internal
23:47:03	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-t7kc5 to ip-10-0-138-47.ec2.internal
23:49:01	openshift-ingress		e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl to ip-10-0-138-142.ec2.internal
23:46:32	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kzkjf to ip-10-0-140-219.ec2.internal
23:42:14	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-5rk8q	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5rk8q to ip-10-0-138-47.ec2.internal
23:46:32	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hrcgk to ip-10-0-138-47.ec2.internal
23:47:03	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmrwv to ip-10-0-140-219.ec2.internal
23:48:41	openshift-ingress		e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd to ip-10-0-138-47.ec2.internal
23:47:02	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-hcml7	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hcml7 to ip-10-0-138-47.ec2.internal
23:46:17	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bkn8v to ip-10-0-138-142.ec2.internal
23:46:17	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bdfdz to ip-10-0-138-47.ec2.internal
23:46:17	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7kbkj to ip-10-0-138-142.ec2.internal
23:45:02	openshift-ingress		payload-pre-processing-7b98bb678c-vc4ft	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-vc4ft to ip-10-0-138-142.ec2.internal
23:39:46	openshift-ingress		maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6bg2t to ip-10-0-140-219.ec2.internal
23:35:00	openshift-ingress		router-default-5b5577b589-v98nd	Scheduled	Successfully assigned openshift-ingress/router-default-5b5577b589-v98nd to ip-10-0-138-142.ec2.internal
23:28:12	openshift-ingress		router-default-5b5577b589-v98nd	FailedScheduling	no nodes available to schedule pods
23:28:10	openshift-ingress		router-default-5b5577b589-v98nd	FailedScheduling	no nodes available to schedule pods
23:49:07	openshift-ingress		payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r to ip-10-0-138-47.ec2.internal
23:48:47	openshift-ingress		payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w to ip-10-0-138-47.ec2.internal
23:45:02	openshift-ingress		payload-processing-5f78687d5b-g2n6t	Scheduled	Successfully assigned openshift-ingress/payload-processing-5f78687d5b-g2n6t to ip-10-0-138-47.ec2.internal
23:49:07	openshift-ingress		payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr to ip-10-0-138-47.ec2.internal
23:48:47	openshift-ingress		payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm to ip-10-0-138-47.ec2.internal
23:41:13	openshift-ingress		data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9 to ip-10-0-138-142.ec2.internal
23:39:46	openshift-ingress		openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs to ip-10-0-138-142.ec2.internal
23:41:12	openshift-ingress		kube-auth-proxy-798f575fff-xvqvg	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-798f575fff-xvqvg to ip-10-0-138-47.ec2.internal
23:27:47	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-5b5577b589 from 0 to 1
23:28:04	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
23:28:04	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
23:28:07	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
23:28:10	openshift-ingress	replicaset-controller	router-default-5b5577b589	SuccessfulCreate	Created pod: router-default-5b5577b589-v98nd
23:34:54 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
23:35:08 (x5)	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
23:35:08 (x5)	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
23:35:16 (x4)	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
23:35:20	openshift-ingress	multus	router-default-5b5577b589-v98nd	AddedInterface	Add eth0 [10.132.0.11/23] from ovn-kubernetes
23:35:21	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	Started	Container started
23:35:21	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	Created	Container created
23:35:21	openshift-ingress	kubelet	router-default-5b5577b589-v98nd	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
23:39:39	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
23:39:41	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-48t7g
23:39:41	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-48t7g	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
23:39:41	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
23:39:42	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
23:39:44	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Created	Container created
23:39:44	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.811s (2.811s including waiting). Image size: 185476477 bytes.
23:39:44	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Started	Container started
23:39:45	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
23:39:45	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
23:39:46	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7f5667f86c from 0 to 1
23:39:46	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-65dccbd94c from 0 to 1
23:39:46	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-6bg2t
23:39:46	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
23:39:46	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-65dccbd94c	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs
23:39:47	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
23:39:47	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
23:39:47	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	AddedInterface	Add eth0 [10.134.0.22/23] from ovn-kubernetes
23:39:47	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
23:39:49	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
23:39:50	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	Started	Container started
23:39:50	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	Created	Container created
23:39:50	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-65dccbd94c-74hhs	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.004s (3.004s including waiting). Image size: 183555333 bytes.
23:39:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 7.823s (7.823s including waiting). Image size: 183555333 bytes.
23:40:01	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:40:01	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:40:17	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:40:17	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:41:12	openshift-ingress	replicaset-controller	kube-auth-proxy-798f575fff	SuccessfulCreate	Created pod: kube-auth-proxy-798f575fff-xvqvg
23:41:12	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-798f575fff from 0 to 2
23:41:12	openshift-ingress	replicaset-controller	kube-auth-proxy-798f575fff	SuccessfulCreate	Created pod: kube-auth-proxy-798f575fff-9ktzc
23:41:13	openshift-ingress	multus	kube-auth-proxy-798f575fff-xvqvg	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
23:41:13	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9	Created	Container created
23:41:13	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9	Started	Container started
23:41:13	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-7cfc97f8f7 from 0 to 1
23:41:13	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-xvqvg	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
23:41:13	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-7cfc97f8f7	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9
23:41:13	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:41:13	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-7cfc97f8f7qwtt9	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
23:41:13	openshift-ingress	multus	kube-auth-proxy-798f575fff-9ktzc	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
23:41:13	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-9ktzc	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-9ktzc	Created	Container created
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-9ktzc	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.545s (2.545s including waiting). Image size: 188904677 bytes.
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-xvqvg	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.446s (2.446s including waiting). Image size: 188904677 bytes.
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-9ktzc	Started	Container started
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-xvqvg	Started	Container started
23:41:16	openshift-ingress	kubelet	kube-auth-proxy-798f575fff-xvqvg	Created	Container created
23:41:27	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:41:27	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:41:29	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:41:29	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:41:43	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:41:43	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:42:14	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
23:42:14	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
23:42:14	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	Created	Container created
23:42:14	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-5rk8q
23:42:14	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
23:42:14	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	Started	Container started
23:42:15	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
23:42:15	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-5rk8q	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
23:42:16	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-48t7g	Killing	Stopping container discovery
23:42:16	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-48t7g
23:42:16	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
23:42:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:42:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:42:41	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:42:41	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:43:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Started	Container started
23:43:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Created	Container created
23:43:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:43:17 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:43:17 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:43:19 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	ProbeError	Startup probe error: Get "http://10.134.0.22:15021/healthz/ready": dial tcp 10.134.0.22:15021: connect: connection refused body:
23:43:19 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Unhealthy	Startup probe failed: Get "http://10.134.0.22:15021/healthz/ready": dial tcp 10.134.0.22:15021: connect: connection refused
23:43:47 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:43:47 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:45:02	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	SuccessfulCreate	Created pod: payload-processing-5f78687d5b-g2n6t
23:45:02	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-vc4ft
23:45:02	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5f78687d5b from 0 to 1
23:45:02 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
23:45:02 (x5)	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	FailedCreate	Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
23:45:02	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
23:45:03	openshift-ingress	kubelet	payload-processing-5f78687d5b-g2n6t	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
23:45:03	openshift-ingress	multus	payload-pre-processing-7b98bb678c-vc4ft	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
23:45:03	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-vc4ft	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
23:45:03	openshift-ingress	multus	payload-processing-5f78687d5b-g2n6t	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
23:45:05	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-vc4ft	Created	Container created
23:45:05	openshift-ingress	kubelet	payload-processing-5f78687d5b-g2n6t	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.92s (1.92s including waiting). Image size: 125353965 bytes.
23:45:05	openshift-ingress	kubelet	payload-processing-5f78687d5b-g2n6t	Created	Container created
23:45:05	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-vc4ft	Started	Container started
23:45:05	openshift-ingress	kubelet	payload-processing-5f78687d5b-g2n6t	Started	Container started
23:45:05	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-vc4ft	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.931s (1.931s including waiting). Image size: 125353965 bytes.
23:46:17	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-bdfdz
23:46:17	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-bkn8v
23:46:17	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-7kbkj
23:46:17	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
23:46:17	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7f5667f86c from 1 to 4
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Started	Container started
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Created	Container created
23:46:18	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	AddedInterface	Add eth0 [10.132.0.64/23] from ovn-kubernetes
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Created	Container created
23:46:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Started	Container started
23:46:18	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
23:46:18	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	AddedInterface	Add eth0 [10.132.0.63/23] from ovn-kubernetes
23:46:21	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.727s (2.727s including waiting). Image size: 183555333 bytes.
23:46:21	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Created	Container created
23:46:21	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Started	Container started
23:46:25 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	ProbeError	Startup probe error: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused body:
23:46:25 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Unhealthy	Startup probe failed: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused
23:46:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	ProbeError	Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused body:
23:46:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Unhealthy	Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused
23:46:28 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	ProbeError	Startup probe error: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused body:
23:46:28 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Unhealthy	Startup probe failed: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused
23:46:32	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-hrcgk
23:46:32	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-kzkjf
23:46:32	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-rrg7c
23:46:32	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 7; reason: cpu resource utilization (percentage of request) above target
23:46:32	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7f5667f86c from 4 to 7
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Started	Container started
23:46:33	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Created	Container created
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Started	Container started
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Created	Container created
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:46:33	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Started	Container started
23:46:33	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
23:46:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Created	Container created
23:46:40 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Unhealthy	Startup probe failed: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused
23:46:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	ProbeError	Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused body:
23:46:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Unhealthy	Startup probe failed: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused
23:46:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	ProbeError	Startup probe error: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused body:
23:46:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Unhealthy	Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused
23:46:40 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	ProbeError	Startup probe error: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused body:
23:47:02	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
23:47:02	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-7f5667f86c from 7 to 10
23:47:02	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-hcml7
23:47:02	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-7f5667f86c-hmrwv
23:47:02	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-7f5667f86c	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-7f5667f86c-t7kc5
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	Created	Container created
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	Started	Container started
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Started	Container started
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Created	Container created
23:47:03	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Started	Container started
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Created	Container created
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:47:03	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:47:03	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:47:03	openshift-ingress	multus	maas-default-gateway-openshift-default-7f5667f86c-hcml7	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	Unhealthy	Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	ProbeError	Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused body:
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Unhealthy	Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Unhealthy	Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	ProbeError	Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused body:
23:47:11 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	ProbeError	Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused body:
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	Unhealthy	Readiness probe failed: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hcml7	ProbeError	Readiness probe error: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:48:13 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	ProbeError	Readiness probe error: Get "http://10.132.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:48:13 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-7kbkj	Unhealthy	Readiness probe failed: Get "http://10.132.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	ProbeError	Readiness probe error: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-t7kc5	Unhealthy	Readiness probe failed: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	Unhealthy	Readiness probe failed: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:48:13 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hmrwv	ProbeError	Readiness probe error: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:48:41	openshift-ingress	deployment-controller	e2e-shared-a-main-8d2ae1-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c from 0 to 1
23:48:41	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-8d2ae1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-8d2ae1
23:48:41	openshift-ingress	replicaset-controller	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c	SuccessfulCreate	Created pod: e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd
23:48:42	openshift-ingress	multus	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
23:48:47 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-8d2ae1: serviceaccount "payload-processing-e2e-shared-a-main-8d2ae1" not found
23:48:47	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w
23:48:47	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-8d2ae1	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699 from 0 to 1
23:48:47	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-8d2ae1	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466 from 0 to 1
23:48:47 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-8d2ae1: serviceaccount "payload-processing-e2e-shared-a-main-8d2ae1" not found
23:48:47	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm
23:48:48	openshift-ingress	multus	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
23:48:48	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
23:48:48	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:48:48	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	Created	Container created
23:48:48	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	Started	Container started
23:48:48	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:48:48	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	Created	Container created
23:48:48	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	Started	Container started
23:49:00	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Unhealthy	Readiness probe failed: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:49:00	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	ProbeError	Readiness probe error: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:49:01	openshift-ingress	replicaset-controller	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c	SuccessfulCreate	Created pod: e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl
23:49:01	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-cd9099	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-cd9099
23:49:01	openshift-ingress	deployment-controller	e2e-shared-b-main-cd9099-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-cd9099-openshift-default-6f9968b77c from 0 to 1
23:49:02	openshift-ingress	multus	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	AddedInterface	Add eth0 [10.132.0.74/23] from ovn-kubernetes
23:49:02 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Started	Container started
23:49:02 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:49:02 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Created	Container created
23:49:07	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccf	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr
23:49:07 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Unhealthy	Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused
23:49:07 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	ProbeError	Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused body:
23:49:07 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-cd9099: serviceaccount "payload-processing-e2e-shared-b-main-cd9099" not found
23:49:07 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccf	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-cd9099: serviceaccount "payload-processing-e2e-shared-b-main-cd9099" not found
23:49:07	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:49:07	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
23:49:07	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-cd9099	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4 from 0 to 1
23:49:07	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-cd9099	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccf from 0 to 1
23:49:07	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r
23:49:08	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:49:08	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Created	Container created
23:49:08	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Started	Container started
23:49:08	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Created	Container created
23:49:08	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Started	Container started
23:49:08	openshift-ingress	multus	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
23:49:23 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Started	Container started
23:49:23 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Created	Container created
23:49:23 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:49:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-8d2ae1-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:49:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-8d2ae1-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:49:28 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	ProbeError	Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused body:
23:49:28 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Unhealthy	Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused
23:50:01 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-cd9099-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:50:01 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-cd9099-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:50:27 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:50:27 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	ProbeError	Readiness probe error: Get "http://10.133.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:50:27 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	Unhealthy	Readiness probe failed: Get "http://10.132.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:50:27 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-rrg7c	ProbeError	Readiness probe error: Get "http://10.132.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:50:27 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-kzkjf	ProbeError	Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:50:27 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-hrcgk	Unhealthy	Readiness probe failed: Get "http://10.133.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:50:28 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	ProbeError	Readiness probe error: Get "http://10.132.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:50:28 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bkn8v	Unhealthy	Readiness probe failed: Get "http://10.132.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:52:45 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	Unhealthy	Readiness probe failed: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:52:45 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-bdfdz	ProbeError	Readiness probe error: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:53:06 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	ProbeError	Readiness probe error: Get "http://10.134.0.22:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:53:06 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-7f5667f86c-6bg2t	Unhealthy	Readiness probe failed: Get "http://10.134.0.22:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:53:17	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-8d2ae1-7f88fcf466-q965w	Killing	Stopping container payload-processing
23:53:17	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-8d2ae1-6d5b87d699cbrzm	Killing	Stopping container payload-pre-processing
23:53:29	openshift-ingress	kubelet	e2e-shared-a-main-8d2ae1-openshift-default-6ff89779c-lcbfd	Killing	Stopping container istio-proxy
23:53:40	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Killing	Stopping container payload-pre-processing
23:53:40	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Killing	Stopping container payload-processing
23:53:49	openshift-ingress	kubelet	e2e-shared-b-main-cd9099-openshift-default-6f9968b77c-42krl	Killing	Stopping container istio-proxy
23:53:59 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-cd9099-568bcd6bf4-spd5r	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
23:53:59 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-cd9099-655dc67ccft6ppr	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")