Events

Time	Namespace	Component	RelatedObject	Reason	Message
05:44:42	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vlqqg to ip-10-0-131-217.ec2.internal
05:43:14	openshift-ingress		payload-pre-processing-7b98bb678c-rkrq6	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-rkrq6 to ip-10-0-141-106.ec2.internal
05:39:00	openshift-ingress		kube-auth-proxy-fbbfd64db-wb5pb	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-fbbfd64db-wb5pb to ip-10-0-131-217.ec2.internal
05:45:12	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b8n6r to ip-10-0-131-217.ec2.internal
05:37:40	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-dqs4f to ip-10-0-136-193.ec2.internal
05:39:00	openshift-ingress		kube-auth-proxy-fbbfd64db-9k5h9	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-fbbfd64db-9k5h9 to ip-10-0-141-106.ec2.internal
05:44:42	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fjdwm to ip-10-0-136-193.ec2.internal
05:45:12	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fztxk to ip-10-0-136-193.ec2.internal
05:44:27	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-kljtq to ip-10-0-141-106.ec2.internal
05:46:44	openshift-ingress		e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf to ip-10-0-141-106.ec2.internal
05:44:27	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-knbs8 to ip-10-0-131-217.ec2.internal
05:45:27	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-krd6s to ip-10-0-141-106.ec2.internal
05:44:42	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-tjlqj to ip-10-0-141-106.ec2.internal
05:44:27	openshift-ingress		maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vv8nk to ip-10-0-141-106.ec2.internal
05:40:06	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-4bvwk	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bvwk to ip-10-0-141-106.ec2.internal
05:37:35	openshift-ingress		istiod-openshift-gateway-7876bfcb74-2hxzv	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-2hxzv to ip-10-0-141-106.ec2.internal
05:37:40	openshift-ingress		openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6 to ip-10-0-136-193.ec2.internal
05:39:00	openshift-ingress		data-science-gateway-data-science-gateway-class-6b748b6585hgmsl	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585hgmsl to ip-10-0-136-193.ec2.internal
05:30:30	openshift-ingress		router-default-7b87875b49-r6w79	Scheduled	Successfully assigned openshift-ingress/router-default-7b87875b49-r6w79 to ip-10-0-136-193.ec2.internal
05:16:16	openshift-ingress		router-default-7b87875b49-r6w79	FailedScheduling	no nodes available to schedule pods
05:16:14	openshift-ingress		router-default-7b87875b49-r6w79	FailedScheduling	no nodes available to schedule pods
05:47:08	openshift-ingress		payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km to ip-10-0-141-106.ec2.internal
05:46:49	openshift-ingress		payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk to ip-10-0-131-217.ec2.internal
05:43:14	openshift-ingress		payload-processing-5f78687d5b-ntlhm	Scheduled	Successfully assigned openshift-ingress/payload-processing-5f78687d5b-ntlhm to ip-10-0-141-106.ec2.internal
05:47:08	openshift-ingress		payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t to ip-10-0-141-106.ec2.internal
05:46:49	openshift-ingress		payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p to ip-10-0-141-106.ec2.internal
05:47:03	openshift-ingress		e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g to ip-10-0-131-217.ec2.internal
05:15:38	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-7b87875b49 from 0 to 1
05:15:48	openshift-ingress	replicaset-controller	router-default-7b87875b49	FailedCreate	Error creating: pods "router-default-7b87875b49-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range
05:16:13	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
05:16:13	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
05:16:14	openshift-ingress	replicaset-controller	router-default-7b87875b49	SuccessfulCreate	Created pod: router-default-7b87875b49-r6w79
05:16:16	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
05:30:26 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
05:30:37 (x5)	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
05:30:37 (x5)	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
05:30:44 (x3)	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
05:30:46	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	Started	Container started
05:30:46	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
05:30:46	openshift-ingress	kubelet	router-default-7b87875b49-r6w79	Created	Container created
05:30:46	openshift-ingress	multus	router-default-7b87875b49-r6w79	AddedInterface	Add eth0 [10.132.0.22/23] from ovn-kubernetes
05:37:34	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
05:37:35	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
05:37:35	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-2hxzv
05:37:36	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-2hxzv	AddedInterface	Add eth0 [10.133.0.17/23] from ovn-kubernetes
05:37:36	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
05:37:38	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.4s (2.4s including waiting). Image size: 185476477 bytes.
05:37:38	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Created	Container created
05:37:38	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Started	Container started
05:37:40	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-dqs4f
05:37:40	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75fd9f5d47 from 0 to 1
05:37:40 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
05:37:40 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
05:37:40	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-6b78c66f8c	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6
05:37:40	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-6b78c66f8c from 0 to 1
05:37:40	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
05:37:41	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
05:37:41	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
05:37:41	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
05:37:41	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
05:37:44	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.26s (3.26s including waiting). Image size: 183555333 bytes.
05:37:44	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.356s (3.356s including waiting). Image size: 183555333 bytes.
05:37:44	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
05:37:45	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	Created	Container created
05:37:45	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-6b78c66f8c-rkpw6	Started	Container started
05:38:05	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:38:05	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:38:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:38:26 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:39:00	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-6b748b6585 from 0 to 1
05:39:00	openshift-ingress	replicaset-controller	kube-auth-proxy-fbbfd64db	SuccessfulCreate	Created pod: kube-auth-proxy-fbbfd64db-9k5h9
05:39:00	openshift-ingress	multus	kube-auth-proxy-fbbfd64db-9k5h9	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
05:39:00	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-6b748b6585	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-6b748b6585hgmsl
05:39:00	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-fbbfd64db from 0 to 2
05:39:00	openshift-ingress	replicaset-controller	kube-auth-proxy-fbbfd64db	SuccessfulCreate	Created pod: kube-auth-proxy-fbbfd64db-wb5pb
05:39:01	openshift-ingress	multus	kube-auth-proxy-fbbfd64db-wb5pb	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
05:39:01	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-6b748b6585hgmsl	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
05:39:01	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6b748b6585hgmsl	Created	Container created
05:39:01	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-9k5h9	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
05:39:01	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-wb5pb	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
05:39:01	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6b748b6585hgmsl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:39:01	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6b748b6585hgmsl	Started	Container started
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-9k5h9	Started	Container started
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-9k5h9	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.551s (2.551s including waiting). Image size: 188904677 bytes.
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-wb5pb	Started	Container started
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-wb5pb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.39s (2.39s including waiting). Image size: 188904677 bytes.
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-wb5pb	Created	Container created
05:39:03	openshift-ingress	kubelet	kube-auth-proxy-fbbfd64db-9k5h9	Created	Container created
05:39:30 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:39:30 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:39:33 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:39:33 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:40:06	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-4bvwk
05:40:06	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-4bvwk	Created	Container created
05:40:06	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-4bvwk	Started	Container started
05:40:06	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-4bvwk	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
05:40:06	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-4bvwk	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
05:40:06	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
05:40:08	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
05:40:08	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-2hxzv
05:40:08	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-2hxzv	Killing	Stopping container discovery
05:40:36 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:40:36 (x3)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:41:04 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Created	Container created
05:41:04 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Started	Container started
05:41:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:41:09 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	ProbeError	Startup probe error: Get "http://10.132.0.32:15021/healthz/ready": dial tcp 10.132.0.32:15021: connect: connection refused body:
05:41:09 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Unhealthy	Startup probe failed: Get "http://10.132.0.32:15021/healthz/ready": dial tcp 10.132.0.32:15021: connect: connection refused
05:41:26 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:41:26 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:41:41 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:41:41 (x3)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:43:14	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-rkrq6
05:43:14	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
05:43:14	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	SuccessfulCreate	Created pod: payload-processing-5f78687d5b-ntlhm
05:43:14	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-5f78687d5b from 0 to 1
05:43:14 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
05:43:14 (x6)	openshift-ingress	replicaset-controller	payload-processing-5f78687d5b	FailedCreate	Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
05:43:15	openshift-ingress	multus	payload-processing-5f78687d5b-ntlhm	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
05:43:15	openshift-ingress	kubelet	payload-processing-5f78687d5b-ntlhm	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
05:43:15	openshift-ingress	multus	payload-pre-processing-7b98bb678c-rkrq6	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
05:43:15	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-rkrq6	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
05:43:17	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-rkrq6	Started	Container started
05:43:17	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-rkrq6	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.238s (2.238s including waiting). Image size: 125317101 bytes.
05:43:17	openshift-ingress	kubelet	payload-processing-5f78687d5b-ntlhm	Started	Container started
05:43:17	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-rkrq6	Created	Container created
05:43:17	openshift-ingress	kubelet	payload-processing-5f78687d5b-ntlhm	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.167s (2.167s including waiting). Image size: 125317101 bytes.
05:43:17	openshift-ingress	kubelet	payload-processing-5f78687d5b-ntlhm	Created	Container created
05:44:27	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
05:44:27	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-vv8nk
05:44:27	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
05:44:27	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
05:44:27	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
05:44:27	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
05:44:27	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-kljtq
05:44:27	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75fd9f5d47 from 1 to 4
05:44:27	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
05:44:27	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
05:44:27	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-knbs8
05:44:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.378s (2.378s including waiting). Image size: 183555333 bytes.
05:44:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Created	Container created
05:44:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Started	Container started
05:44:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Started	Container started
05:44:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Created	Container created
05:44:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.428s (3.428s including waiting). Image size: 183555333 bytes.
05:44:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Started	Container started
05:44:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Created	Container created
05:44:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 5.925s (5.925s including waiting). Image size: 183555333 bytes.
05:44:37 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	ProbeError	Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused body:
05:44:37 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Unhealthy	Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused
05:44:38 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Unhealthy	Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused
05:44:38 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	ProbeError	Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused body:
05:44:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Unhealthy	Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused
05:44:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	ProbeError	Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused body:
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Started	Container started
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Created	Container created
05:44:42	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75fd9f5d47 from 4 to 7
05:44:42	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 7; reason:
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Created	Container created
05:44:42	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
05:44:42	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-tjlqj
05:44:42	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Started	Container started
05:44:42	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-fjdwm
05:44:42	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-vlqqg
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Started	Container started
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Created	Container created
05:44:42	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:44:42	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	ProbeError	Startup probe error: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused body:
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Unhealthy	Startup probe failed: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	ProbeError	Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused body:
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	ProbeError	Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused body:
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Unhealthy	Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused
05:44:49 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Unhealthy	Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused
05:44:52 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Unhealthy	Startup probe failed: Get "http://10.132.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:44:52 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	ProbeError	Startup probe error: Get "http://10.132.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:44:53 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Unhealthy	Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:44:53 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	ProbeError	Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:44:53 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Unhealthy	Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:44:53 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	ProbeError	Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:45:12	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 9; reason: cpu resource utilization (percentage of request) above target
05:45:12	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75fd9f5d47 from 7 to 9
05:45:12	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-b8n6r
05:45:12	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
05:45:12	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75fd9f5d47-fztxk
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Started	Container started
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:45:13	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Created	Container created
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	Created	Container created
05:45:13	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	Started	Container started
05:45:20 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Unhealthy	Startup probe failed: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused
05:45:20 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	ProbeError	Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused body:
05:45:20 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-b8n6r	Unhealthy	Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused
05:45:20 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	ProbeError	Startup probe error: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused body:
05:45:27	openshift-ingress	multus	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
05:45:27	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
05:45:27	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75fd9f5d47 from 9 to 10
05:45:27	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75fd9f5d47	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-75fd9f5d47-krd6s
05:45:27	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:45:28	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Started	Container started
05:45:28	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Created	Container created
05:45:35 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Unhealthy	Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused
05:45:35 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	ProbeError	Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused body:
05:46:07	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	ProbeError	Readiness probe error: Get "http://10.132.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:46:07	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fztxk	Unhealthy	Readiness probe failed: Get "http://10.132.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:46:07	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	ProbeError	Readiness probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:46:07	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-krd6s	Unhealthy	Readiness probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:46:28	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	ProbeError	Readiness probe error: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:46:28	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-kljtq	Unhealthy	Readiness probe failed: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:46:43	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-3493b9	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-3493b9
05:46:44	openshift-ingress	replicaset-controller	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75	SuccessfulCreate	Created pod: e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf
05:46:44	openshift-ingress	multus	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
05:46:44	openshift-ingress	deployment-controller	e2e-shared-a-main-3493b9-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-3493b9-openshift-default-649fb9cc75 from 0 to 1
05:46:49 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-3493b9: serviceaccount "payload-processing-e2e-shared-a-main-3493b9" not found
05:46:49	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk
05:46:49	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-3493b9	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-3493b9-78ff5f8944 from 0 to 1
05:46:49	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-3493b9	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879 from 0 to 1
05:46:49	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p
05:46:49 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-3493b9: serviceaccount "payload-processing-e2e-shared-a-main-3493b9" not found
05:46:50	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
05:46:50	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
05:46:50	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
05:46:50	openshift-ingress	multus	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
05:46:50	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	Created	Container created
05:46:50	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	Started	Container started
05:46:52	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Started	Container started
05:46:52	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Created	Container created
05:46:52	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.98s (1.98s including waiting). Image size: 125317101 bytes.
05:47:02	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Unhealthy	Readiness probe failed: Get "http://10.133.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:47:02	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	ProbeError	Readiness probe error: Get "http://10.133.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:47:03	openshift-ingress	deployment-controller	e2e-shared-b-main-84403a-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-84403a-openshift-default-59975dfc57 from 0 to 1
05:47:03	openshift-ingress	replicaset-controller	e2e-shared-b-main-84403a-openshift-default-59975dfc57	SuccessfulCreate	Created pod: e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g
05:47:03	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-84403a	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-84403a
05:47:04 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Started	Container started
05:47:04 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Created	Container created
05:47:04	openshift-ingress	multus	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
05:47:04 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:47:08	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-84403a-547759fcdb	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km
05:47:08 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-84403a: serviceaccount "payload-processing-e2e-shared-b-main-84403a" not found
05:47:08	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-84403a	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c from 0 to 1
05:47:08	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-84403a	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-84403a-547759fcdb from 0 to 1
05:47:08	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t
05:47:08 (x6)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-84403a-547759fcdb	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-84403a-547759fcdb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-84403a: serviceaccount "payload-processing-e2e-shared-b-main-84403a" not found
05:47:09	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Started	Container started
05:47:09	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Started	Container started
05:47:09	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Created	Container created
05:47:09 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	ProbeError	Startup probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused body:
05:47:09 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Unhealthy	Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused
05:47:09	openshift-ingress	multus	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
05:47:09	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
05:47:09	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
05:47:09	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Created	Container created
05:47:09	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
05:47:22	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	ProbeError	Readiness probe error: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:47:22	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Unhealthy	Readiness probe failed: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:47:24 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Started	Container started
05:47:24 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Created	Container created
05:47:24 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
05:47:29 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-3493b9-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:47:29 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	ProbeError	Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused body:
05:47:29 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-3493b9-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:47:29 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Unhealthy	Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused
05:47:44	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-3493b9-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:47:44	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-3493b9-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
05:47:48 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-84403a-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:47:48 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-84403a-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
05:50:24	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:50:24	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vlqqg	ProbeError	Readiness probe error: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:50:25 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	ProbeError	Readiness probe error: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:50:25 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-knbs8	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:50:41 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	Unhealthy	Readiness probe failed: Get "http://10.132.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:50:41 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-dqs4f	ProbeError	Readiness probe error: Get "http://10.132.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:51:09 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	Unhealthy	Readiness probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:51:09 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-tjlqj	ProbeError	Readiness probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:51:09 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	ProbeError	Readiness probe error: Get "http://10.132.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:51:09 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	ProbeError	Readiness probe error: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
05:51:09 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-vv8nk	Unhealthy	Readiness probe failed: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:51:09 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75fd9f5d47-fjdwm	Unhealthy	Readiness probe failed: Get "http://10.132.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
05:51:15	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-3493b9-78ff5f8944-pt4wk	Killing	Stopping container payload-processing
05:51:16	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-3493b9-7df4bc879-qrx6p	Killing	Stopping container payload-pre-processing
05:51:28	openshift-ingress	kubelet	e2e-shared-a-main-3493b9-openshift-default-649fb9cc75-5nrrf	Killing	Stopping container istio-proxy
05:51:39	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Killing	Stopping container payload-processing
05:51:39	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Killing	Stopping container payload-pre-processing
05:51:47	openshift-ingress	kubelet	e2e-shared-b-main-84403a-openshift-default-59975dfc57-zdw7g	Killing	Stopping container istio-proxy
05:52:01 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-84403a-547759fcdb-gn8km	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
05:52:01 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-84403a-86f57cf8c-h2s8t	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")