Events

Time	Namespace	Component	RelatedObject	Reason	Message
09:04:16	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dr8ns to ip-10-0-140-65.ec2.internal
08:57:25	openshift-ingress		istiod-openshift-gateway-7876bfcb74-97klg	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-97klg to ip-10-0-135-148.ec2.internal
09:05:01	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-9p2l5	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-9p2l5 to ip-10-0-140-65.ec2.internal
08:58:47	openshift-ingress		kube-auth-proxy-7466c8b6f6-9vsk8	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-7466c8b6f6-9vsk8 to ip-10-0-138-30.ec2.internal
08:58:47	openshift-ingress		kube-auth-proxy-7466c8b6f6-87f2z	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-7466c8b6f6-87f2z to ip-10-0-135-148.ec2.internal
09:04:31	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-hw27k	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hw27k to ip-10-0-135-148.ec2.internal
09:04:01	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4hc8w to ip-10-0-135-148.ec2.internal
08:57:30	openshift-ingress		openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc to ip-10-0-140-65.ec2.internal
08:59:52	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cdrr6 to ip-10-0-140-65.ec2.internal
09:06:31	openshift-ingress		e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p to ip-10-0-140-65.ec2.internal
09:06:37	openshift-ingress		payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7 to ip-10-0-140-65.ec2.internal
09:06:56	openshift-ingress		payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw to ip-10-0-140-65.ec2.internal
09:05:01	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-tk78f	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tk78f to ip-10-0-138-30.ec2.internal
09:03:01	openshift-ingress		payload-processing-5f78687d5b-j2vnb	Scheduled	Successfully assigned openshift-ingress/payload-processing-5f78687d5b-j2vnb to ip-10-0-140-65.ec2.internal
09:04:01	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rfbgz to ip-10-0-138-30.ec2.internal
09:04:31	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-p5khn	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p5khn to ip-10-0-140-65.ec2.internal
09:04:16	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-48clp	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-48clp to ip-10-0-135-148.ec2.internal
08:57:30	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wkmvv to ip-10-0-140-65.ec2.internal
09:03:02	openshift-ingress		payload-pre-processing-7b98bb678c-w7gvc	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-w7gvc to ip-10-0-140-65.ec2.internal
08:58:48	openshift-ingress		data-science-gateway-data-science-gateway-class-7c689579c-cm45f	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cm45f to ip-10-0-140-65.ec2.internal
09:06:37	openshift-ingress		payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f to ip-10-0-140-65.ec2.internal
09:06:57	openshift-ingress		payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7 to ip-10-0-140-65.ec2.internal
08:52:54	openshift-ingress		router-default-777d6b9d77-br8dn	Scheduled	Successfully assigned openshift-ingress/router-default-777d6b9d77-br8dn to ip-10-0-140-65.ec2.internal
08:46:36	openshift-ingress		router-default-777d6b9d77-br8dn	FailedScheduling	no nodes available to schedule pods
09:06:51	openshift-ingress		e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2 to ip-10-0-135-148.ec2.internal
08:46:34	openshift-ingress		router-default-777d6b9d77-br8dn	FailedScheduling	no nodes available to schedule pods
09:04:16	openshift-ingress		maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ncsq9 to ip-10-0-138-30.ec2.internal
08:46:08	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-777d6b9d77 from 0 to 1
08:46:17	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
08:46:17	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
08:46:21	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
08:46:34	openshift-ingress	replicaset-controller	router-default-777d6b9d77	SuccessfulCreate	Created pod: router-default-777d6b9d77-br8dn
08:52:39 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
08:53:02 (x5)	openshift-ingress	kubelet	router-default-777d6b9d77-br8dn	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
08:53:02 (x5)	openshift-ingress	kubelet	router-default-777d6b9d77-br8dn	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
08:53:07	openshift-ingress	kubelet	router-default-777d6b9d77-br8dn	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
08:53:07	openshift-ingress	kubelet	router-default-777d6b9d77-br8dn	Started	Container started
08:53:07	openshift-ingress	multus	router-default-777d6b9d77-br8dn	AddedInterface	Add eth0 [10.132.0.17/23] from ovn-kubernetes
08:53:07	openshift-ingress	kubelet	router-default-777d6b9d77-br8dn	Created	Container created
08:57:24	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
08:57:25	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
08:57:25	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-97klg
08:57:26	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-97klg	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
08:57:26	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-97klg	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
08:57:29	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-97klg	Started	Container started
08:57:29	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-97klg	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.969s (2.969s including waiting). Image size: 185476477 bytes.
08:57:29	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-97klg	Created	Container created
08:57:30	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-74987c9b47 from 0 to 1
08:57:30	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
08:57:30	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-wkmvv
08:57:30	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
08:57:30	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-fcb75c45f from 0 to 1
08:57:30	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-74987c9b47	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc
08:57:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
08:57:31	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
08:57:31	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
08:57:33	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
08:57:39	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.393s (8.393s including waiting). Image size: 183555333 bytes.
08:57:39	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.672s (8.672s including waiting). Image size: 183555333 bytes.
08:57:40	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	Created	Container created
08:57:40	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-74987c9b47-r7mpc	Started	Container started
08:57:55	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:57:55	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
08:58:15 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:58:15 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:58:47	openshift-ingress	replicaset-controller	kube-auth-proxy-7466c8b6f6	SuccessfulCreate	Created pod: kube-auth-proxy-7466c8b6f6-9vsk8
08:58:47	openshift-ingress	replicaset-controller	kube-auth-proxy-7466c8b6f6	SuccessfulCreate	Created pod: kube-auth-proxy-7466c8b6f6-87f2z
08:58:47	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-7466c8b6f6 from 0 to 2
08:58:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7c689579c-cm45f	Started	Container started
08:58:48	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-7c689579c from 0 to 1
08:58:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7c689579c-cm45f	Created	Container created
08:58:48	openshift-ingress	multus	kube-auth-proxy-7466c8b6f6-9vsk8	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
08:58:48	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-9vsk8	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
08:58:48	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-87f2z	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
08:58:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7c689579c-cm45f	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
08:58:48	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-7c689579c-cm45f	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
08:58:48	openshift-ingress	multus	kube-auth-proxy-7466c8b6f6-87f2z	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
08:58:48	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-7c689579c	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-7c689579c-cm45f
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-9vsk8	Created	Container created
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-9vsk8	Started	Container started
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-9vsk8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.362s (2.362s including waiting). Image size: 188904677 bytes.
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-87f2z	Started	Container started
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-87f2z	Created	Container created
08:58:50	openshift-ingress	kubelet	kube-auth-proxy-7466c8b6f6-87f2z	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.373s (2.373s including waiting). Image size: 188904677 bytes.
08:59:33 (x3)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:59:33 (x3)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:59:35 (x3)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:59:35 (x3)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
08:59:52	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
08:59:52	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-cdrr6
08:59:53	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
08:59:53	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
08:59:56	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Started	Container started
08:59:56	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Created	Container created
08:59:56	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.684s (2.684s including waiting). Image size: 185476477 bytes.
08:59:57	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
08:59:57	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-cdrr6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
08:59:58	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-97klg
08:59:58	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-97klg	Killing	Stopping container discovery
08:59:58	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
09:00:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:00:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:01:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:01:10 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Started	Container started
09:01:10 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Created	Container created
09:01:15 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	ProbeError	Startup probe error: Get "http://10.132.0.35:15021/healthz/ready": dial tcp 10.132.0.35:15021: connect: connection refused body:
09:01:15 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-wkmvv	Unhealthy	Startup probe failed: Get "http://10.132.0.35:15021/healthz/ready": dial tcp 10.132.0.35:15021: connect: connection refused
09:01:31 (x4)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:01:31 (x4)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:01:46 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
09:01:46 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
09:03:01 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
09:03:01	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
09:03:01 (x5)	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	FailedCreate	Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
09:03:01	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	SuccessfulCreate	Created pod: payload-processing-5f78687d5b-j2vnb
09:03:01	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5f78687d5b from 0 to 1
09:03:02	openshift-ingress	multus	payload-pre-processing-7b98bb678c-w7gvc	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
09:03:02	openshift-ingress	kubelet	payload-processing-5f78687d5b-j2vnb	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
09:03:02	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-w7gvc
09:03:02	openshift-ingress	multus	payload-processing-5f78687d5b-j2vnb	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
09:03:02	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-w7gvc	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
09:03:04	openshift-ingress	kubelet	payload-processing-5f78687d5b-j2vnb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.231s (2.231s including waiting). Image size: 125317101 bytes.
09:03:04	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-w7gvc	Started	Container started
09:03:04	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-w7gvc	Created	Container created
09:03:04	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-w7gvc	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.246s (2.246s including waiting). Image size: 125317101 bytes.
09:03:04	openshift-ingress	kubelet	payload-processing-5f78687d5b-j2vnb	Started	Container started
09:03:04	openshift-ingress	kubelet	payload-processing-5f78687d5b-j2vnb	Created	Container created
09:04:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-4hc8w
09:04:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-rfbgz
09:04:01	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 3; reason: cpu resource utilization (percentage of request) above target
09:04:01	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-fcb75c45f from 1 to 3
09:04:02	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
09:04:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
09:04:02	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
09:04:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.273s (3.273s including waiting). Image size: 183555333 bytes.
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.071s (3.071s including waiting). Image size: 183555333 bytes.
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Started	Container started
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Created	Container created
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Started	Container started
09:04:05	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Created	Container created
09:04:11 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Unhealthy	Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused
09:04:11 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	ProbeError	Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused body:
09:04:12 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	ProbeError	Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused body:
09:04:12 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Unhealthy	Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused
09:04:16	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-ncsq9
09:04:16	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 6; reason: cpu resource utilization (percentage of request) above target
09:04:16	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-fcb75c45f from 3 to 6
09:04:16	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-dr8ns
09:04:16	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-48clp
09:04:17	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Created	Container created
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Started	Container started
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	Started	Container started
09:04:17	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Created	Container created
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Started	Container started
09:04:17	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-48clp	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:04:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	Created	Container created
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	Unhealthy	Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	ProbeError	Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused body:
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Unhealthy	Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	ProbeError	Startup probe error: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused body:
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Unhealthy	Startup probe failed: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused
09:04:23 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	ProbeError	Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused body:
09:04:31	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason:
09:04:31	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-hw27k
09:04:31	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-fcb75c45f from 6 to 8
09:04:31	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-p5khn
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Created	Container created
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Started	Container started
09:04:32	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-p5khn	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Started	Container started
09:04:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Created	Container created
09:04:32	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-hw27k	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
09:04:38 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Unhealthy	Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused
09:04:38 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	ProbeError	Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused body:
09:04:38 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Unhealthy	Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused
09:04:38 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	ProbeError	Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused body:
09:04:43 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Unhealthy	Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:04:43 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	ProbeError	Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:04:43 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	ProbeError	Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:04:43 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Unhealthy	Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:05:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-fcb75c45f-tk78f
09:05:01	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-fcb75c45f	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-fcb75c45f-9p2l5
09:05:01	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
09:05:01	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-fcb75c45f from 8 to 10
09:05:02	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-tk78f	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-tk78f	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:05:02	openshift-ingress	multus	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	AddedInterface	Add eth0 [10.132.0.63/23] from ovn-kubernetes
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	Created	Container created
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	Started	Container started
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-tk78f	Created	Container created
09:05:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-tk78f	Started	Container started
09:05:09 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-tk78f	Unhealthy	Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused
09:05:09 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-tk78f	ProbeError	Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused body:
09:05:10 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	ProbeError	Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused body:
09:05:10 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-9p2l5	Unhealthy	Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused
09:05:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	Unhealthy	Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:05:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	ProbeError	Readiness probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:05:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-48clp	Unhealthy	Readiness probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:05:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-ncsq9	ProbeError	Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:06:31	openshift-ingress	replicaset-controller	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5	SuccessfulCreate	Created pod: e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p
09:06:31	openshift-ingress	deployment-controller	e2e-shared-a-main-7bd398-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-7bd398-openshift-default-76b656ffb5 from 0 to 1
09:06:31	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-7bd398	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-7bd398
09:06:32	openshift-ingress	multus	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	AddedInterface	Add eth0 [10.132.0.69/23] from ovn-kubernetes
09:06:37	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7
09:06:37 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-7bd398: serviceaccount "payload-processing-e2e-shared-a-main-7bd398" not found
09:06:37	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f
09:06:37	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-7bd398	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495 from 0 to 1
09:06:37	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-7bd398	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d from 0 to 1
09:06:37 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-7bd398: serviceaccount "payload-processing-e2e-shared-a-main-7bd398" not found
09:06:38	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	Created	Container created
09:06:38	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	Started	Container started
09:06:38	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	AddedInterface	Add eth0 [10.132.0.70/23] from ovn-kubernetes
09:06:38	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
09:06:38	openshift-ingress	multus	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	AddedInterface	Add eth0 [10.132.0.71/23] from ovn-kubernetes
09:06:38	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	Started	Container started
09:06:38	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
09:06:38	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	Created	Container created
09:06:50	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	ProbeError	Readiness probe error: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:06:50	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Unhealthy	Readiness probe failed: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:06:51	openshift-ingress	replicaset-controller	e2e-shared-b-main-3f394b-openshift-default-5597bb8887	SuccessfulCreate	Created pod: e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2
09:06:51	openshift-ingress	deployment-controller	e2e-shared-b-main-3f394b-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-3f394b-openshift-default-5597bb8887 from 0 to 1
09:06:51	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-3f394b	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-3f394b
09:06:52 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Created	Container created
09:06:52 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:06:52 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Started	Container started
09:06:52	openshift-ingress	multus	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
09:06:56	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-3f394b	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8 from 0 to 1
09:06:56 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-3f394b: serviceaccount "payload-processing-e2e-shared-b-main-3f394b" not found
09:06:56	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-3f394b	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff from 0 to 1
09:06:56	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw
09:06:56 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-3f394b: serviceaccount "payload-processing-e2e-shared-b-main-3f394b" not found
09:06:56	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7
09:06:57	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Created	Container created
09:06:57	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
09:06:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Started	Container started
09:06:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
09:06:57	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Started	Container started
09:06:57	openshift-ingress	multus	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	AddedInterface	Add eth0 [10.132.0.73/23] from ovn-kubernetes
09:06:57	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	AddedInterface	Add eth0 [10.132.0.72/23] from ovn-kubernetes
09:06:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Created	Container created
09:06:57 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	ProbeError	Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused body:
09:06:57 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Unhealthy	Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused
09:07:10	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Unhealthy	Readiness probe failed: Get "http://10.133.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:07:10	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	ProbeError	Readiness probe error: Get "http://10.133.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:07:11 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Started	Container started
09:07:11 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Created	Container created
09:07:11 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
09:07:16 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Unhealthy	Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused
09:07:16 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	ProbeError	Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused body:
09:07:31 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-7bd398-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:07:31 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-7bd398-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:07:36 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-3f394b-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:07:36 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-3f394b-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
09:08:26 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	Unhealthy	Readiness probe failed: Get "http://10.132.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:08:26 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-dr8ns	ProbeError	Readiness probe error: Get "http://10.132.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:10:44 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	ProbeError	Readiness probe error: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:10:44 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	Unhealthy	Readiness probe failed: Get "http://10.132.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:10:44 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	ProbeError	Readiness probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:10:44 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-hw27k	Unhealthy	Readiness probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:10:44 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-rfbgz	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:10:44 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-p5khn	ProbeError	Readiness probe error: Get "http://10.132.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:10:45 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	ProbeError	Readiness probe error: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
09:10:45 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-fcb75c45f-4hc8w	Unhealthy	Readiness probe failed: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
09:11:08	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-7bd398-58947ff495wtps7	Killing	Stopping container payload-pre-processing
09:11:08	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-7bd398-5cf77c9b9d-mqb7f	Killing	Stopping container payload-processing
09:11:20	openshift-ingress	kubelet	e2e-shared-a-main-7bd398-openshift-default-76b656ffb5-g6j7p	Killing	Stopping container istio-proxy
09:11:31	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Killing	Stopping container payload-processing
09:11:31	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Killing	Stopping container payload-pre-processing
09:11:40	openshift-ingress	kubelet	e2e-shared-b-main-3f394b-openshift-default-5597bb8887-ncsp2	Killing	Stopping container istio-proxy
09:11:49 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-3f394b-7f4bb44bff55whw	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
09:11:49 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-3f394b-5c9786f8d8-2jjs7	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")