Events

Time	Namespace	Component	RelatedObject	Reason	Message
17:29:17	kube-system		kas-connection-checker-685895c7c9-l5cs4	Scheduled	Successfully assigned kube-system/kas-connection-checker-685895c7c9-l5cs4 to ip-10-0-133-117.us-east-2.compute.internal
17:42:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-dpcwb	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dpcwb to ip-10-0-130-81.us-east-2.compute.internal
17:28:59	kube-system		konnectivity-agent-79hsd	Scheduled	Successfully assigned kube-system/konnectivity-agent-79hsd to ip-10-0-130-81.us-east-2.compute.internal
17:40:20	openshift-ingress		payload-processing-55fbbbf9dc-qq4w2	Scheduled	Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-qq4w2 to ip-10-0-141-130.us-east-2.compute.internal
17:41:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-58ghk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-58ghk to ip-10-0-141-130.us-east-2.compute.internal
17:30:43	default	apiserver	kube-system	TerminationGracefulTerminationFinished	All pending requests processed
17:29:43	default	apiserver	kube-system	TerminationPreShutdownHooksFinished	All pre-shutdown hooks have been finished
17:29:43	default	apiserver	kube-system	TerminationStoppedServing	Server has stopped listening
17:29:43	default	apiserver	kube-system	TerminationMinimalShutdownDurationFinished	The minimal shutdown duration of 15s finished
17:29:17	openshift-ingress		router-default-5488654f9f-t5cnd	Scheduled	Successfully assigned openshift-ingress/router-default-5488654f9f-t5cnd to ip-10-0-133-117.us-east-2.compute.internal
17:35:38	default		default-dsc	ProvisioningError	failed to lookup object /default-kserve: no matches for kind "Kserve" in version "components.platform.opendatahub.io/v1alpha1"
17:46:10	openshift-ingress		e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp to ip-10-0-141-130.us-east-2.compute.internal
17:29:28	default	apiserver	kube-system	TerminationStart	Received signal to terminate, becoming unready, but keeping serving
17:41:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-hzgtt	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-hzgtt to ip-10-0-133-117.us-east-2.compute.internal
17:41:37	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-wh8tk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wh8tk to ip-10-0-141-130.us-east-2.compute.internal
17:41:37	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-5jvbr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5jvbr to ip-10-0-130-81.us-east-2.compute.internal
17:35:38	openshift-ingress		kube-auth-proxy-77d87c9759-tfvjn	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-77d87c9759-tfvjn to ip-10-0-130-81.us-east-2.compute.internal
17:46:46	openshift-ingress		payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk to ip-10-0-133-117.us-east-2.compute.internal
17:35:38	openshift-ingress		kube-auth-proxy-77d87c9759-b8z5w	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-77d87c9759-b8z5w to ip-10-0-141-130.us-east-2.compute.internal
17:23:53	openshift-ingress		router-default-5488654f9f-t5cnd	FailedScheduling	no nodes available to schedule pods
17:23:55	openshift-ingress		router-default-5488654f9f-t5cnd	FailedScheduling	no nodes available to schedule pods
17:46:15	openshift-ingress		payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg to ip-10-0-133-117.us-east-2.compute.internal
17:41:37	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-pdhst	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdhst to ip-10-0-141-130.us-east-2.compute.internal
17:46:46	openshift-ingress		payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx to ip-10-0-133-117.us-east-2.compute.internal
17:34:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-jq5zt	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jq5zt to ip-10-0-133-117.us-east-2.compute.internal
17:28:59	kube-system		global-pull-secret-syncer-g5wh5	Scheduled	Successfully assigned kube-system/global-pull-secret-syncer-g5wh5 to ip-10-0-130-81.us-east-2.compute.internal
17:28:48	kube-system		global-pull-secret-syncer-5g66m	Scheduled	Successfully assigned kube-system/global-pull-secret-syncer-5g66m to ip-10-0-133-117.us-east-2.compute.internal
17:42:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-j5vsb	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-j5vsb to ip-10-0-141-130.us-east-2.compute.internal
17:46:42	openshift-ingress		e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr to ip-10-0-141-130.us-east-2.compute.internal
17:28:48	kube-system		global-pull-secret-syncer-j72rg	Scheduled	Successfully assigned kube-system/global-pull-secret-syncer-j72rg to ip-10-0-141-130.us-east-2.compute.internal
17:29:17	kube-system		kas-connection-checker-685895c7c9-v7699	Scheduled	Successfully assigned kube-system/kas-connection-checker-685895c7c9-v7699 to ip-10-0-133-117.us-east-2.compute.internal
17:34:22	openshift-ingress		openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n to ip-10-0-133-117.us-east-2.compute.internal
17:23:36	kube-system		kas-connection-checker-685895c7c9-l5cs4	FailedScheduling	no nodes available to schedule pods
17:41:37	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-tqfmm	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tqfmm to ip-10-0-133-117.us-east-2.compute.internal
17:23:36	kube-system		kas-connection-checker-685895c7c9-4rjfs	FailedScheduling	no nodes available to schedule pods
17:40:20	openshift-ingress		payload-pre-processing-7b98bb678c-qdds7	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-qdds7 to ip-10-0-141-130.us-east-2.compute.internal
17:28:48	kube-system		konnectivity-agent-rzndd	Scheduled	Successfully assigned kube-system/konnectivity-agent-rzndd to ip-10-0-141-130.us-east-2.compute.internal
17:23:34	kube-system		kas-connection-checker-685895c7c9-4rjfs	FailedScheduling	no nodes available to schedule pods
17:36:37	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-sll8p	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sll8p to ip-10-0-130-81.us-east-2.compute.internal
17:29:17	kube-system		kas-connection-checker-685895c7c9-4rjfs	Scheduled	Successfully assigned kube-system/kas-connection-checker-685895c7c9-4rjfs to ip-10-0-133-117.us-east-2.compute.internal
17:34:17	openshift-ingress		istiod-openshift-gateway-7876bfcb74-b5lfs	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5lfs to ip-10-0-130-81.us-east-2.compute.internal
17:23:34	kube-system		kas-connection-checker-685895c7c9-l5cs4	FailedScheduling	no nodes available to schedule pods
17:35:38	openshift-ingress		data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl to ip-10-0-133-117.us-east-2.compute.internal
17:28:48	kube-system		konnectivity-agent-4x8wd	Scheduled	Successfully assigned kube-system/konnectivity-agent-4x8wd to ip-10-0-133-117.us-east-2.compute.internal
17:35:39	default		default-dsc	ProvisioningError	failed to lookup object /default-kserve: no matches for kind "Kserve" in version "components.platform.opendatahub.io/v1alpha1"
17:35:40	default		default-dsc	ProvisioningError	failed to lookup object /default-kserve: no matches for kind "Kserve" in version "components.platform.opendatahub.io/v1alpha1"
17:35:41	default		default-dsc	ProvisioningError	failed to lookup object /default-kserve: no matches for kind "Kserve" in version "components.platform.opendatahub.io/v1alpha1"
17:35:39	default		default-gateway	ProvisioningError	failed to detect ingress mode: failed to update GatewayConfig with detected mode: Operation cannot be fulfilled on gatewayconfigs.services.platform.opendatahub.io "default-gateway": the object has been modified; please apply your changes to the latest version and try again
17:41:22	openshift-ingress		maas-default-gateway-openshift-default-75d7c785f-68x67	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68x67 to ip-10-0-130-81.us-east-2.compute.internal
17:23:34	kube-system		kas-connection-checker-685895c7c9-v7699	FailedScheduling	no nodes available to schedule pods
17:23:36	kube-system		kas-connection-checker-685895c7c9-v7699	FailedScheduling	no nodes available to schedule pods
17:46:15	openshift-ingress		payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb to ip-10-0-133-117.us-east-2.compute.internal
17:22:34	default	apiserver	openshift-kube-apiserver	KubeAPIReadyz	readyz=true
17:22:47	kube-system	default-scheduler	kube-scheduler	LeaderElection	kube-scheduler-9f99dd475-pzhgg_9fb6fe3d-95dd-4f1d-ab57-796ed68d24bd became leader
17:22:50	kube-system	kube-controller-manager	kube-controller-manager	LeaderElection	kube-controller-manager-74c6bbd6f7-v6hbv_ffce8392-563f-4e20-a892-c83c8afd545e became leader
17:23:07	openshift-cluster-version	openshift-cluster-version	version	LeaderElection	cluster-version-operator-dc4fbc7c6-vfph2_d58ce6b7-f59d-4d38-ac8e-a467a78f281a became leader
17:23:09	openshift-cluster-version	openshift-cluster-version	version	LoadPayload	Loading payload version="4.22.20" image="registry.build12.ci.openshift.org/ci-op-6kp6bi09/release@sha256:b94e4b85832b43dfa35785551effe5d16e122e3fa3ed277b6d765c945fe5ef14"
17:23:09	openshift-cluster-version	openshift-cluster-version	version	RetrievePayload	Retrieving and verifying payload version="4.22.20" image="registry.build12.ci.openshift.org/ci-op-6kp6bi09/release@sha256:b94e4b85832b43dfa35785551effe5d16e122e3fa3ed277b6d765c945fe5ef14"
17:23:10	openshift-cluster-version	openshift-cluster-version	version	PayloadLoaded	Payload loaded version="4.22.20" image="registry.build12.ci.openshift.org/ci-op-6kp6bi09/release@sha256:b94e4b85832b43dfa35785551effe5d16e122e3fa3ed277b6d765c945fe5ef14" architecture="Multi"
17:23:34	kube-system	deployment-controller	kas-connection-checker	ScalingReplicaSet	Scaled up replica set kas-connection-checker-685895c7c9 from 0 to 3
17:23:34	kube-system	replicaset-controller	kas-connection-checker-685895c7c9	SuccessfulCreate	Created pod: kas-connection-checker-685895c7c9-4rjfs
17:23:34	kube-system	replicaset-controller	kas-connection-checker-685895c7c9	SuccessfulCreate	Created pod: kas-connection-checker-685895c7c9-l5cs4
17:23:34	kube-system	replicaset-controller	kas-connection-checker-685895c7c9	SuccessfulCreate	Created pod: kas-connection-checker-685895c7c9-v7699
17:23:40	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-5488654f9f from 0 to 1
17:23:50	openshift-ingress	replicaset-controller	router-default-5488654f9f	FailedCreate	Error creating: pods "router-default-5488654f9f-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range
17:23:53	openshift-ingress	replicaset-controller	router-default-5488654f9f	SuccessfulCreate	Created pod: router-default-5488654f9f-t5cnd
17:24:17	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
17:24:17	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
17:24:21	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
17:24:22	kube-system	kube-controller-manager	kube-controller-manager	LeaderElection	kube-controller-manager-6b8c44fc86-d8bzp_1a9808d0-9323-4128-8252-b71a6274a731 became leader
17:28:47 (x7)	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasSufficientPID	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasSufficientPID
17:28:47 (x7)	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasNoDiskPressure	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
17:28:47	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeAllocatableEnforced	Updated limits on system reserved cgroup /system.slice
17:28:47	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeAllocatableEnforced	Updated Node Allocatable limit across pods
17:28:48	default	node-controller	ip-10-0-133-117.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-133-117.us-east-2.compute.internal event: Registered Node ip-10-0-133-117.us-east-2.compute.internal in Controller
17:28:48	default	cloud-node-controller	ip-10-0-141-130.us-east-2.compute.internal	Synced	Node synced successfully
17:28:48	default	node-controller	ip-10-0-141-130.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-141-130.us-east-2.compute.internal event: Registered Node ip-10-0-141-130.us-east-2.compute.internal in Controller
17:28:48	kube-system	daemonset-controller	global-pull-secret-syncer	SuccessfulCreate	Created pod: global-pull-secret-syncer-j72rg
17:28:48	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-141-130.us-east-2.compute.internal	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e"
17:28:48	kube-system	daemonset-controller	konnectivity-agent	SuccessfulCreate	Created pod: konnectivity-agent-4x8wd
17:28:48	default	cloud-node-controller	ip-10-0-133-117.us-east-2.compute.internal	Synced	Node synced successfully
17:28:48 (x8)	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasSufficientMemory	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasSufficientMemory
17:28:48	kube-system	daemonset-controller	konnectivity-agent	SuccessfulCreate	Created pod: konnectivity-agent-rzndd
17:28:48	kube-system	daemonset-controller	global-pull-secret-syncer	SuccessfulCreate	Created pod: global-pull-secret-syncer-5g66m
17:28:48 (x7)	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasSufficientPID	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasSufficientPID
17:28:48 (x7)	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasNoDiskPressure	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
17:28:48 (x7)	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasSufficientMemory	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasSufficientMemory
17:28:49	kube-system	kubelet	konnectivity-agent-rzndd	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406"
17:28:49	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-133-117.us-east-2.compute.internal	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e"
17:28:50	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-133-117.us-east-2.compute.internal	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" in 1.664s (1.664s including waiting). Image size: 294084393 bytes.
17:28:50	kube-system	kubelet	konnectivity-agent-4x8wd	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406"
17:28:50	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-133-117.us-east-2.compute.internal	Created	Container created
17:28:50	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-133-117.us-east-2.compute.internal	Started	Container started
17:28:51	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-141-130.us-east-2.compute.internal	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" in 2.522s (2.522s including waiting). Image size: 294084393 bytes.
17:28:51	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-141-130.us-east-2.compute.internal	Started	Container started
17:28:51	kube-system	kubelet	kube-apiserver-proxy-ip-10-0-141-130.us-east-2.compute.internal	Created	Container created
17:28:54	kube-system	kubelet	konnectivity-agent-rzndd	Created	Container created
17:28:54	kube-system	kubelet	konnectivity-agent-rzndd	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406" in 4.711s (4.711s including waiting). Image size: 276927292 bytes.
17:28:54	kube-system	kubelet	konnectivity-agent-4x8wd	Started	Container started
17:28:54	kube-system	kubelet	konnectivity-agent-rzndd	Started	Container started
17:28:54	kube-system	kubelet	konnectivity-agent-4x8wd	Created	Container created
17:28:54	kube-system	kubelet	konnectivity-agent-4x8wd	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406" in 4.175s (4.175s including waiting). Image size: 276927292 bytes.
17:28:59	kube-system	daemonset-controller	global-pull-secret-syncer	SuccessfulCreate	Created pod: global-pull-secret-syncer-g5wh5
17:28:59	default	cloud-node-controller	ip-10-0-130-81.us-east-2.compute.internal	Synced	Node synced successfully
17:28:59	kube-system	daemonset-controller	konnectivity-agent	SuccessfulCreate	Created pod: konnectivity-agent-79hsd
17:29:00 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
17:29:01	kube-system	kubelet	konnectivity-agent-79hsd	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406"
17:29:03	default	node-controller	ip-10-0-130-81.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-130-81.us-east-2.compute.internal event: Registered Node ip-10-0-130-81.us-east-2.compute.internal in Controller
17:29:04	kube-system	kubelet	konnectivity-agent-79hsd	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406" in 3.71s (3.71s including waiting). Image size: 276927292 bytes.
17:29:05 (x6)	kube-system	kubelet	global-pull-secret-syncer-j72rg	FailedMount	MountVolume.SetUp failed for volume "original-pull-secret" : object "kube-system"/"original-pull-secret" not registered
17:29:05 (x6)	kube-system	kubelet	global-pull-secret-syncer-5g66m	FailedMount	MountVolume.SetUp failed for volume "original-pull-secret" : object "kube-system"/"original-pull-secret" not registered
17:29:05 (x2)	kube-system	kubelet	konnectivity-agent-79hsd	Failed	Error: services have not yet been read at least once, cannot construct envvars
17:29:10 (x7)	default	ovnk-controlplane	ip-10-0-133-117.us-east-2.compute.internal	ErrorReconcilingNode	[k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-133-117.us-east-2.compute.internal, error getting gateway config for node ip-10-0-133-117.us-east-2.compute.internal: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-133-117.us-east-2.compute.internal", failed to update chassis to local for local node ip-10-0-133-117.us-east-2.compute.internal, error: failed to parse node chassis-id for node - ip-10-0-133-117.us-east-2.compute.internal, error: k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-133-117.us-east-2.compute.internal]
17:29:10 (x8)	default	ovnk-controlplane	ip-10-0-141-130.us-east-2.compute.internal	ErrorReconcilingNode	[k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-141-130.us-east-2.compute.internal, error getting gateway config for node ip-10-0-141-130.us-east-2.compute.internal: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-141-130.us-east-2.compute.internal", failed to update chassis to local for local node ip-10-0-141-130.us-east-2.compute.internal, error: failed to parse node chassis-id for node - ip-10-0-141-130.us-east-2.compute.internal, error: k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-141-130.us-east-2.compute.internal]
17:29:15	kube-system	kubelet	konnectivity-agent-79hsd	Created	Container created
17:29:15 (x2)	kube-system	kubelet	konnectivity-agent-79hsd	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406" already present on machine and can be accessed by the pod
17:29:16 (x16)	kube-system	kubelet	global-pull-secret-syncer-j72rg	NetworkNotReady	network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
17:29:16	kube-system	kubelet	konnectivity-agent-79hsd	Started	Container started
17:29:16 (x6)	kube-system	kubelet	global-pull-secret-syncer-g5wh5	FailedMount	MountVolume.SetUp failed for volume "original-pull-secret" : object "kube-system"/"original-pull-secret" not registered
17:29:17	kube-system	multus	kas-connection-checker-685895c7c9-4rjfs	AddedInterface	Add eth0 [10.133.0.6/23] from ovn-kubernetes
17:29:17	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeReady	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeReady
17:29:17	kube-system	kubelet	kas-connection-checker-685895c7c9-4rjfs	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472" already present on machine and can be accessed by the pod
17:29:17 (x15)	kube-system	kubelet	global-pull-secret-syncer-5g66m	NetworkNotReady	network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
17:29:17	kube-system	multus	kas-connection-checker-685895c7c9-v7699	AddedInterface	Add eth0 [10.133.0.8/23] from ovn-kubernetes
17:29:17	kube-system	kubelet	kas-connection-checker-685895c7c9-l5cs4	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472" already present on machine and can be accessed by the pod
17:29:17	kube-system	multus	kas-connection-checker-685895c7c9-l5cs4	AddedInterface	Add eth0 [10.133.0.7/23] from ovn-kubernetes
17:29:17	kube-system	kubelet	kas-connection-checker-685895c7c9-v7699	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472" already present on machine and can be accessed by the pod
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-l5cs4	Created	Container created
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-v7699	Created	Container created
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-v7699	Started	Container started
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-l5cs4	Started	Container started
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-4rjfs	Started	Container started
17:29:18	kube-system	kubelet	kas-connection-checker-685895c7c9-4rjfs	Created	Container created
17:29:20 (x7)	default	ovnk-controlplane	ip-10-0-130-81.us-east-2.compute.internal	ErrorReconcilingNode	[k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-130-81.us-east-2.compute.internal, error getting gateway config for node ip-10-0-130-81.us-east-2.compute.internal: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-0-130-81.us-east-2.compute.internal", failed to update chassis to local for local node ip-10-0-130-81.us-east-2.compute.internal, error: failed to parse node chassis-id for node - ip-10-0-130-81.us-east-2.compute.internal, error: k8s.ovn.org/node-chassis-id annotation not found for node ip-10-0-130-81.us-east-2.compute.internal]
17:29:21	kube-system	multus	global-pull-secret-syncer-j72rg	AddedInterface	Add eth0 [10.132.0.4/23] from ovn-kubernetes
17:29:21	kube-system	kubelet	global-pull-secret-syncer-j72rg	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87"
17:29:22	kube-system	multus	global-pull-secret-syncer-5g66m	AddedInterface	Add eth0 [10.133.0.4/23] from ovn-kubernetes
17:29:22	kube-system	kubelet	global-pull-secret-syncer-5g66m	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87"
17:29:25 (x14)	kube-system	kubelet	global-pull-secret-syncer-g5wh5	NetworkNotReady	network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
17:29:25 (x5)	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
17:29:25 (x5)	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
17:29:27	kube-system	kubelet	global-pull-secret-syncer-j72rg	Started	Container started
17:29:27	kube-system	kubelet	global-pull-secret-syncer-j72rg	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87" in 6.214s (6.214s including waiting). Image size: 627364882 bytes.
17:29:27	kube-system	kubelet	global-pull-secret-syncer-j72rg	Created	Container created
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasSufficientPID	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasSufficientPID
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasNoDiskPressure	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasSufficientPID	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasSufficientPID
17:29:28	kube-system	kubelet	global-pull-secret-syncer-5g66m	Started	Container started
17:29:28	kube-system	kubelet	global-pull-secret-syncer-5g66m	Created	Container created
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	Starting	Starting kubelet.
17:29:28	kube-system	kubelet	global-pull-secret-syncer-5g66m	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87" in 6.17s (6.17s including waiting). Image size: 627364882 bytes.
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasSufficientMemory	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasSufficientMemory
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeAllocatableEnforced	Updated limits on system reserved cgroup /system.slice
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeAllocatableEnforced	Updated Node Allocatable limit across pods
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	Starting	Starting kubelet.
17:29:28	default	kubelet	ip-10-0-141-130.us-east-2.compute.internal	NodeHasNoDiskPressure	Node ip-10-0-141-130.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeHasSufficientMemory	Node ip-10-0-133-117.us-east-2.compute.internal status is now: NodeHasSufficientMemory
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeAllocatableEnforced	Updated limits on system reserved cgroup /system.slice
17:29:28	default	kubelet	ip-10-0-133-117.us-east-2.compute.internal	NodeAllocatableEnforced	Updated Node Allocatable limit across pods
17:29:30 (x2)	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
17:29:32	kube-system	multus	global-pull-secret-syncer-g5wh5	AddedInterface	Add eth0 [10.134.0.5/23] from ovn-kubernetes
17:29:33 (x4)	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
17:29:33	kube-system	kubelet	global-pull-secret-syncer-g5wh5	Pulling	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87"
17:29:37	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	Created	Container created
17:29:37	openshift-ingress	multus	router-default-5488654f9f-t5cnd	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
17:29:37	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
17:29:37	openshift-ingress	kubelet	router-default-5488654f9f-t5cnd	Started	Container started
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	NodeHasNoDiskPressure	Node ip-10-0-130-81.us-east-2.compute.internal status is now: NodeHasNoDiskPressure
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	Starting	Starting kubelet.
17:29:38	kube-system	kubelet	global-pull-secret-syncer-g5wh5	Created	Container created
17:29:38	kube-system	kubelet	global-pull-secret-syncer-g5wh5	Pulled	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe8e89e6cbec3b50985683a4c052c0353f3b869fd22dae8191f84dbdcb591f87" in 5.127s (5.127s including waiting). Image size: 627364882 bytes.
17:29:38	kube-system	kubelet	global-pull-secret-syncer-g5wh5	Started	Container started
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	NodeHasSufficientPID	Node ip-10-0-130-81.us-east-2.compute.internal status is now: NodeHasSufficientPID
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	NodeHasSufficientMemory	Node ip-10-0-130-81.us-east-2.compute.internal status is now: NodeHasSufficientMemory
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	NodeAllocatableEnforced	Updated Node Allocatable limit across pods
17:29:38	default	kubelet	ip-10-0-130-81.us-east-2.compute.internal	NodeAllocatableEnforced	Updated limits on system reserved cgroup /system.slice
17:29:49	default	node-controller	ip-10-0-130-81.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-130-81.us-east-2.compute.internal event: Registered Node ip-10-0-130-81.us-east-2.compute.internal in Controller
17:29:49	default	node-controller	ip-10-0-133-117.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-133-117.us-east-2.compute.internal event: Registered Node ip-10-0-133-117.us-east-2.compute.internal in Controller
17:29:49	kube-system	kube-controller-manager	kube-controller-manager	LeaderElection	kube-controller-manager-74bb96b5b7-mcj6l_684ec695-a93a-4694-902a-1531a26fc656 became leader
17:29:49	default	node-controller	ip-10-0-141-130.us-east-2.compute.internal	RegisteredNode	Node ip-10-0-141-130.us-east-2.compute.internal event: Registered Node ip-10-0-141-130.us-east-2.compute.internal in Controller
17:34:15	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
17:34:17	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-b5lfs
17:34:17	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-b5lfs	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
17:34:17	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-b5lfs	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
17:34:17	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
17:34:21	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-b5lfs	Started	Container started
17:34:21	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-b5lfs	Created	Container created
17:34:21	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-b5lfs	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.235s (3.235s including waiting). Image size: 185476477 bytes.
17:34:22	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-5cf4665f7f from 0 to 1
17:34:22	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-5cf4665f7f	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n
17:34:22	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75d7c785f from 0 to 1
17:34:22	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-jq5zt	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
17:34:22	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
17:34:22	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
17:34:22	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
17:34:22	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
17:34:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-jq5zt
17:34:24	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
17:34:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 7.883s (7.883s including waiting). Image size: 183555333 bytes.
17:34:30	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 7.968s (7.968s including waiting). Image size: 183555333 bytes.
17:34:31	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	Created	Container created
17:34:31	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-5cf4665f7f-z9h7n	Started	Container started
17:35:07 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:07 (x3)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:22	kube-system	cert-manager-cainjector-5c57bbd4fc-fxnnt_1b2383ba-b97d-4043-88ec-ee4bafba7441	cert-manager-cainjector-leader-election	LeaderElection	cert-manager-cainjector-5c57bbd4fc-fxnnt_1b2383ba-b97d-4043-88ec-ee4bafba7441 became leader
17:35:38	openshift-ingress	replicaset-controller	kube-auth-proxy-77d87c9759	SuccessfulCreate	Created pod: kube-auth-proxy-77d87c9759-b8z5w
17:35:38	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-84d4d45d7c from 0 to 1
17:35:38	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-84d4d45d7c	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl
17:35:38	openshift-ingress	replicaset-controller	kube-auth-proxy-77d87c9759	SuccessfulCreate	Created pod: kube-auth-proxy-77d87c9759-tfvjn
17:35:38	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-77d87c9759 from 0 to 2
17:35:39	openshift-ingress	multus	kube-auth-proxy-77d87c9759-tfvjn	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
17:35:39	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl	Started	Container started
17:35:39	openshift-ingress	multus	kube-auth-proxy-77d87c9759-b8z5w	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
17:35:39	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
17:35:39	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:35:39	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-84d4d45d7cfglwl	Created	Container created
17:35:39	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-tfvjn	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
17:35:39	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-b8z5w	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
17:35:41	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-tfvjn	Created	Container created
17:35:41	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-tfvjn	Started	Container started
17:35:41	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-tfvjn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.407s (2.407s including waiting). Image size: 188904677 bytes.
17:35:41	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-b8z5w	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.774s (2.774s including waiting). Image size: 188904677 bytes.
17:35:42	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-b8z5w	Created	Container created
17:35:42	openshift-ingress	kubelet	kube-auth-proxy-77d87c9759-b8z5w	Started	Container started
17:35:56	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:56	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:56	kube-system	cert-manager-leader-election	cert-manager-controller	LeaderElection	cert-manager-597b95cb84-ww6nh-external-cert-manager-controller became leader
17:36:08 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:08 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:11	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:11	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:37	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
17:36:37	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-sll8p
17:36:38	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-sll8p	Started	Container started
17:36:38	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-sll8p	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
17:36:38	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-sll8p	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
17:36:38	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-sll8p	Created	Container created
17:36:39	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-b5lfs	Killing	Stopping container discovery
17:36:39	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
17:36:39	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-b5lfs
17:36:47 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:47 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:37:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:37:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:38:01	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:38:01 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Started	Container started
17:38:01 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Created	Container created
17:38:07 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:38:07 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:38:07 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Unhealthy	Startup probe failed: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused
17:38:07 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	ProbeError	Startup probe error: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused body:
17:38:22 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:38:22 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:40:19	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1
17:40:20	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-qdds7	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
17:40:20 (x5)	openshift-ingress	replicaset-controller	payload-processing-55fbbbf9dc	FailedCreate	Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
17:40:20	openshift-ingress	replicaset-controller	payload-processing-55fbbbf9dc	SuccessfulCreate	Created pod: payload-processing-55fbbbf9dc-qq4w2
17:40:20	openshift-ingress	kubelet	payload-processing-55fbbbf9dc-qq4w2	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
17:40:20	openshift-ingress	multus	payload-processing-55fbbbf9dc-qq4w2	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
17:40:20	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
17:40:20	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-qdds7
17:40:20	openshift-ingress	multus	payload-pre-processing-7b98bb678c-qdds7	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
17:40:20 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
17:40:23	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-qdds7	Created	Container created
17:40:23	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-qdds7	Started	Container started
17:40:23	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-qdds7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.215s (2.215s including waiting). Image size: 125353963 bytes.
17:40:23	openshift-ingress	kubelet	payload-processing-55fbbbf9dc-qq4w2	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.23s (2.23s including waiting). Image size: 125353963 bytes.
17:40:23	openshift-ingress	kubelet	payload-processing-55fbbbf9dc-qq4w2	Created	Container created
17:40:23	openshift-ingress	kubelet	payload-processing-55fbbbf9dc-qq4w2	Started	Container started
17:41:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-58ghk
17:41:22	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75d7c785f from 1 to 4
17:41:22	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
17:41:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-68x67
17:41:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-hzgtt
17:41:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	Started	Container started
17:41:23	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-hzgtt	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
17:41:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
17:41:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	Created	Container created
17:41:23	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-58ghk	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
17:41:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
17:41:23	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-68x67	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
17:41:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.093s (3.093s including waiting). Image size: 183555333 bytes.
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Started	Container started
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Created	Container created
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Created	Container created
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.698s (2.698s including waiting). Image size: 183555333 bytes.
17:41:26	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Started	Container started
17:41:30 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	Unhealthy	Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused
17:41:30 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	ProbeError	Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused body:
17:41:33 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Unhealthy	Startup probe failed: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused
17:41:33 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	ProbeError	Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused body:
17:41:33 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Unhealthy	Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused
17:41:33 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	ProbeError	Startup probe error: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused body:
17:41:37	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-wh8tk
17:41:37	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-tqfmm
17:41:37	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-5jvbr
17:41:37	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-pdhst
17:41:37	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason:
17:41:37	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75d7c785f from 4 to 8
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	Started	Container started
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:41:38	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-5jvbr	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	Created	Container created
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	Started	Container started
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	Started	Container started
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	Created	Container created
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:41:38	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-wh8tk	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	Created	Container created
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:41:38	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-tqfmm	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	Started	Container started
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	Created	Container created
17:41:38	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:41:38	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-pdhst	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	ProbeError	Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused body:
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	Unhealthy	Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	Unhealthy	Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	ProbeError	Startup probe error: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused body:
17:41:45 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	Unhealthy	Startup probe failed: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused
17:41:45 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	ProbeError	Startup probe error: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused body:
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	Unhealthy	Startup probe failed: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused
17:41:45 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	ProbeError	Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused body:
17:42:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-75d7c785f-j5vsb
17:42:22	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
17:42:22	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-75d7c785f from 8 to 10
17:42:22	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-75d7c785f	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-75d7c785f-dpcwb
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:42:23	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-dpcwb	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	Started	Container started
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	Started	Container started
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	Created	Container created
17:42:23	openshift-ingress	multus	maas-default-gateway-openshift-default-75d7c785f-j5vsb	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
17:42:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	Created	Container created
17:42:31 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	Unhealthy	Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused
17:42:31 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	ProbeError	Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused body:
17:42:31 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	ProbeError	Startup probe error: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused body:
17:42:31 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	Unhealthy	Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused
17:45:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	ProbeError	Readiness probe error: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:45:17	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-tqfmm	Unhealthy	Readiness probe failed: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:45:32 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:45:32 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-5jvbr	ProbeError	Readiness probe error: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:45:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	ProbeError	Readiness probe error: Get "http://10.133.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:45:32	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-hzgtt	Unhealthy	Readiness probe failed: Get "http://10.133.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:46:10	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-e930f1	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-e930f1
17:46:10	openshift-ingress	replicaset-controller	e2e-shared-a-main-e930f1-openshift-default-777d95d795	SuccessfulCreate	Created pod: e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp
17:46:10	openshift-ingress	deployment-controller	e2e-shared-a-main-e930f1-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-e930f1-openshift-default-777d95d795 from 0 to 1
17:46:10	openshift-ingress	multus	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
17:46:15 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-e930f1: serviceaccount "payload-processing-e2e-shared-a-main-e930f1" not found
17:46:15 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-e930f1: serviceaccount "payload-processing-e2e-shared-a-main-e930f1" not found
17:46:15	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-e930f1	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc from 0 to 1
17:46:15	openshift-ingress	multus	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	AddedInterface	Add eth0 [10.133.0.64/23] from ovn-kubernetes
17:46:15	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg
17:46:15	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb
17:46:15	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
17:46:15	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
17:46:15	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-e930f1	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7 from 0 to 1
17:46:16	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
17:46:18	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Created	Container created
17:46:18	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Started	Container started
17:46:18	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Created	Container created
17:46:18	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.568s (2.568s including waiting). Image size: 125353963 bytes.
17:46:18	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.598s (2.598s including waiting). Image size: 125353963 bytes.
17:46:18	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Started	Container started
17:46:27	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Unhealthy	Readiness probe failed: Get "http://10.132.0.55:15021/healthz/ready": read tcp 10.132.0.2:44542->10.132.0.55:15021: read: connection reset by peer
17:46:27	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	ProbeError	Readiness probe error: Get "http://10.132.0.55:15021/healthz/ready": read tcp 10.132.0.2:44542->10.132.0.55:15021: read: connection reset by peer body:
17:46:28 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:46:28 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Created	Container created
17:46:28 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Started	Container started
17:46:34 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	ProbeError	Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused body:
17:46:34 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Unhealthy	Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused
17:46:40 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e930f1-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:40 (x2)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e930f1-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:41	openshift-ingress	replicaset-controller	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55	SuccessfulCreate	Created pod: e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr
17:46:41	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-ed6b15	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-ed6b15
17:46:41	openshift-ingress	deployment-controller	e2e-shared-b-main-ed6b15-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-ed6b15-openshift-default-86c57fff55 from 0 to 1
17:46:42	openshift-ingress	multus	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
17:46:46	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-ed6b15	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4c from 0 to 1
17:46:46 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-ed6b15: serviceaccount "payload-processing-e2e-shared-b-main-ed6b15" not found
17:46:46	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4c	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk
17:46:46 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4c	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-ed6b15: serviceaccount "payload-processing-e2e-shared-b-main-ed6b15" not found
17:46:46	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-ed6b15	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-ed6b15-6799675bf7 from 0 to 1
17:46:46	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx
17:46:47	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Created	Container created
17:46:47	openshift-ingress	multus	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	AddedInterface	Add eth0 [10.133.0.66/23] from ovn-kubernetes
17:46:47	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	AddedInterface	Add eth0 [10.133.0.65/23] from ovn-kubernetes
17:46:47	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
17:46:47	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Started	Container started
17:46:47	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Started	Container started
17:46:47	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Created	Container created
17:46:47	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
17:46:55	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e930f1-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:46:55	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e930f1-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:46:59	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Unhealthy	Readiness probe failed: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:46:59	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	ProbeError	Readiness probe error: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:47:02 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Started	Container started
17:47:02 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Created	Container created
17:47:02 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
17:47:07 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Unhealthy	Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused
17:47:07 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	ProbeError	Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused body:
17:47:42 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ed6b15-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:47:42 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-ed6b15-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:50:09 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	Unhealthy	Readiness probe failed: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:09 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-jq5zt	ProbeError	Readiness probe error: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:32 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	Unhealthy	Readiness probe failed: Get "http://10.132.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:32 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-pdhst	ProbeError	Readiness probe error: Get "http://10.132.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:33 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:33 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	Unhealthy	Readiness probe failed: Get "http://10.132.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:33 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	Unhealthy	Readiness probe failed: Get "http://10.132.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:33 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-j5vsb	ProbeError	Readiness probe error: Get "http://10.132.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:33 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-wh8tk	ProbeError	Readiness probe error: Get "http://10.132.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:33 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-dpcwb	ProbeError	Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:35 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	Unhealthy	Readiness probe failed: Get "http://10.132.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:35 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	Unhealthy	Readiness probe failed: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:50:35 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-68x67	ProbeError	Readiness probe error: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:50:35 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-75d7c785f-58ghk	ProbeError	Readiness probe error: Get "http://10.132.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
17:51:05	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e930f1-d4f78d8cc-8dcpg	Killing	Stopping container payload-processing
17:51:05	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e930f1-74bb7d88b7n7gjb	Killing	Stopping container payload-pre-processing
17:51:14	openshift-ingress	kubelet	e2e-shared-a-main-e930f1-openshift-default-777d95d795-65fqp	Killing	Stopping container istio-proxy
17:51:21	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Killing	Stopping container payload-pre-processing
17:51:21	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Killing	Stopping container payload-processing
17:51:28	openshift-ingress	kubelet	e2e-shared-b-main-ed6b15-openshift-default-86c57fff55-8bmnr	Killing	Stopping container istio-proxy
17:51:39 (x2)	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-ed6b15-78f69b7c4cfm9pk	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
17:51:39 (x2)	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-ed6b15-6799675bf7-8lsxx	Unhealthy	Readiness probe failed: service unhealthy (responded with "NOT_SERVING")