Events

Time	Namespace	Component	RelatedObject	Reason	Message
23:06:06	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-rgdzs	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rgdzs to ip-10-0-141-83.us-east-2.compute.internal
22:55:47	openshift-ingress		kube-auth-proxy-685596c46f-cvm95	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-685596c46f-cvm95 to ip-10-0-138-89.us-east-2.compute.internal
22:55:28	cert-manager		cert-manager-597b95cb84-m9rwx	Scheduled	Successfully assigned cert-manager/cert-manager-597b95cb84-m9rwx to ip-10-0-141-83.us-east-2.compute.internal
23:11:23	llm		e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	Scheduled	Successfully assigned llm/e2e-embedding-simulated-kserve-67bf594cd6-pkjh4 to ip-10-0-138-89.us-east-2.compute.internal
23:11:19	llm		e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	Scheduled	Successfully assigned llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt to ip-10-0-128-124.us-east-2.compute.internal
23:11:26	llm		e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	Scheduled	Successfully assigned llm/e2e-trlp-test-simulated-kserve-69b588897d-bcnjr to ip-10-0-128-124.us-east-2.compute.internal
23:19:12	ai-tenants		e2e-shared-a-main-70a7bf	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-a-main-70a7bf: MaasTenantConfig is not available to restore revocation operands
23:11:28	llm		e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	Scheduled	Successfully assigned llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6 to ip-10-0-138-89.us-east-2.compute.internal
23:11:29	llm		echo-simulator-kserve-64b6f8f96c-6q4sf	Scheduled	Successfully assigned llm/echo-simulator-kserve-64b6f8f96c-6q4sf to ip-10-0-128-124.us-east-2.compute.internal
22:55:21	cert-manager		cert-manager-cainjector-5c57bbd4fc-59v6p	Scheduled	Successfully assigned cert-manager/cert-manager-cainjector-5c57bbd4fc-59v6p to ip-10-0-138-89.us-east-2.compute.internal
22:55:23	openshift-jobset-operator		jobset-operator-674cd59dc5-5s8w9	Scheduled	Successfully assigned openshift-jobset-operator/jobset-operator-674cd59dc5-5s8w9 to ip-10-0-138-89.us-east-2.compute.internal
23:11:31	llm		facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	Scheduled	Successfully assigned llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9d645 to ip-10-0-128-124.us-east-2.compute.internal
22:55:51	openshift-jobset-operator		jobset-controller-manager-7c646f4476-xp7tk	Scheduled	Successfully assigned openshift-jobset-operator/jobset-controller-manager-7c646f4476-xp7tk to ip-10-0-128-124.us-east-2.compute.internal
23:11:32	llm		premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	Scheduled	Successfully assigned llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn to ip-10-0-128-124.us-east-2.compute.internal
22:55:41	kuadrant-system		limitador-operator-controller-manager-5778bb4449-xgxbq	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-5778bb4449-xgxbq to ip-10-0-128-124.us-east-2.compute.internal
23:11:22	llm		e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	Scheduled	Successfully assigned llm/e2e-distinct-simulated-kserve-7b998b79d9-6nbz5 to ip-10-0-138-89.us-east-2.compute.internal
22:56:40	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-hww68	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hww68 to ip-10-0-138-89.us-east-2.compute.internal
22:54:34	openshift-ingress		istiod-openshift-gateway-7876bfcb74-mmgdm	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-mmgdm to ip-10-0-128-124.us-east-2.compute.internal
22:55:17	cert-manager		cert-manager-webhook-5dbbbf75cc-qssgz	Scheduled	Successfully assigned cert-manager/cert-manager-webhook-5dbbbf75cc-qssgz to ip-10-0-141-83.us-east-2.compute.internal
23:14:40	openshift-ingress		e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b to ip-10-0-138-89.us-east-2.compute.internal
23:14:21	openshift-ingress		e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n to ip-10-0-138-89.us-east-2.compute.internal
22:55:47	openshift-ingress		data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2 to ip-10-0-138-89.us-east-2.compute.internal
23:13:58	llm		grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	Scheduled	Successfully assigned llm/grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm to ip-10-0-128-124.us-east-2.compute.internal
23:16:46	llm		grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	Scheduled	Successfully assigned llm/grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk to ip-10-0-141-83.us-east-2.compute.internal
23:10:22	opendatahub		ai-gateway-controller-5d8dddb676-pnbq9	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-5d8dddb676-pnbq9 to ip-10-0-128-124.us-east-2.compute.internal
23:06:26	kuadrant-system		limitador-limitador-7c9b87d574-vf2qv	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-7c9b87d574-vf2qv to ip-10-0-128-124.us-east-2.compute.internal
22:55:35	kuadrant-system		authorino-operator-7b7bb4bc77-fbm7v	Scheduled	Successfully assigned kuadrant-system/authorino-operator-7b7bb4bc77-fbm7v to ip-10-0-128-124.us-east-2.compute.internal
23:10:38	opendatahub		ai-gateway-controller-95895fbfc-2qtm9	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-95895fbfc-2qtm9 to ip-10-0-138-89.us-east-2.compute.internal
22:56:40	opendatahub		kserve-controller-manager-776cc66d47-dfkvs	Scheduled	Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-dfkvs to ip-10-0-141-83.us-east-2.compute.internal
23:08:58	kuadrant-system		authorino-6dd76875b5-jghnb	Scheduled	Successfully assigned kuadrant-system/authorino-6dd76875b5-jghnb to ip-10-0-128-124.us-east-2.compute.internal
22:55:14	opendatahub-operator-system		opendatahub-operator-controller-manager-8b65dd898-w5dkz	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-8b65dd898-w5dkz to ip-10-0-138-89.us-east-2.compute.internal
23:06:25	kuadrant-system		authorino-56c6c9ddd5-cps9r	Scheduled	Successfully assigned kuadrant-system/authorino-56c6c9ddd5-cps9r to ip-10-0-128-124.us-east-2.compute.internal
22:50:42	openshift-ingress		router-default-764475789f-szjgr	Scheduled	Successfully assigned openshift-ingress/router-default-764475789f-szjgr to ip-10-0-141-83.us-east-2.compute.internal
22:43:44	openshift-ingress		router-default-764475789f-szjgr	FailedScheduling	no nodes available to schedule pods
22:43:42	openshift-ingress		router-default-764475789f-szjgr	FailedScheduling	no nodes available to schedule pods
23:14:45	openshift-ingress		payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b to ip-10-0-128-124.us-east-2.compute.internal
23:06:25	kuadrant-system		limitador-limitador-74dfcfb894-dnwfl	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-74dfcfb894-dnwfl to ip-10-0-128-124.us-east-2.compute.internal
23:14:26	openshift-ingress		payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5 to ip-10-0-138-89.us-east-2.compute.internal
23:16:49	llm		grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Scheduled	Successfully assigned llm/grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q to ip-10-0-128-124.us-east-2.compute.internal
23:10:28	openshift-ingress		payload-processing-7bb9489676-59ml8	Scheduled	Successfully assigned openshift-ingress/payload-processing-7bb9489676-59ml8 to ip-10-0-128-124.us-east-2.compute.internal
22:55:50	kuadrant-system		authorino-57f7f8659f-rz2w9	Scheduled	Successfully assigned kuadrant-system/authorino-57f7f8659f-rz2w9 to ip-10-0-128-124.us-east-2.compute.internal
23:14:45	openshift-ingress		payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2 to ip-10-0-128-124.us-east-2.compute.internal
22:55:14	opendatahub-operator-system		opendatahub-operator-controller-manager-8b65dd898-sf2v5	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-8b65dd898-sf2v5 to ip-10-0-141-83.us-east-2.compute.internal
23:14:26	openshift-ingress		payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs to ip-10-0-138-89.us-east-2.compute.internal
23:10:28	openshift-ingress		payload-pre-processing-7b98bb678c-mrg4r	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-mrg4r to ip-10-0-138-89.us-east-2.compute.internal
23:08:15	opendatahub		usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	Scheduled	Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2tgkz to ip-10-0-141-83.us-east-2.compute.internal
22:55:31	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-wc9rp	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wc9rp to ip-10-0-141-83.us-east-2.compute.internal
23:12:38	llm		grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Scheduled	Successfully assigned llm/grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2 to ip-10-0-138-89.us-east-2.compute.internal
23:19:26	ai-tenants		e2e-shared-b-main-8235df	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-b-main-8235df: MaasTenantConfig is not available to restore revocation operands
23:09:46	opendatahub		llmisvc-controller-manager-6466c554d8-fl2c7	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-fl2c7 to ip-10-0-141-83.us-east-2.compute.internal
22:55:49	opendatahub		kserve-module-controller-manager-7566cf5fc8-w29fq	Scheduled	Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-w29fq to ip-10-0-141-83.us-east-2.compute.internal
22:54:53	openshift-ingress		openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b to ip-10-0-138-89.us-east-2.compute.internal
22:55:37	kuadrant-system		kuadrant-console-plugin-66c4fb575d-j5bdq	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-66c4fb575d-j5bdq to ip-10-0-138-89.us-east-2.compute.internal
23:14:12	opendatahub		test-curl-12475-f2203b	Scheduled	Successfully assigned opendatahub/test-curl-12475-f2203b to ip-10-0-138-89.us-east-2.compute.internal
23:09:43	kuadrant-system		authorino-5cd48575b8-lbldd	Scheduled	Successfully assigned kuadrant-system/authorino-5cd48575b8-lbldd to ip-10-0-128-124.us-east-2.compute.internal
23:12:09	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-pf29j to ip-10-0-141-83.us-east-2.compute.internal
23:11:54	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-kjjnb to ip-10-0-138-89.us-east-2.compute.internal
23:12:42	llm		grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	Scheduled	Successfully assigned llm/grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s to ip-10-0-138-89.us-east-2.compute.internal
23:08:58	kuadrant-system		authorino-64545899bb-nsg72	Scheduled	Successfully assigned kuadrant-system/authorino-64545899bb-nsg72 to ip-10-0-128-124.us-east-2.compute.internal
22:54:53	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-jtf6f to ip-10-0-138-89.us-east-2.compute.internal
23:11:54	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-gxm54 to ip-10-0-141-83.us-east-2.compute.internal
23:14:18	opendatahub		test-curl-12475-65b8a9	Scheduled	Successfully assigned opendatahub/test-curl-12475-65b8a9 to ip-10-0-138-89.us-east-2.compute.internal
23:12:09	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-gpnwl to ip-10-0-138-89.us-east-2.compute.internal
23:12:09	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-g9ntf to ip-10-0-128-124.us-east-2.compute.internal
23:13:09	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-ddn95 to ip-10-0-138-89.us-east-2.compute.internal
23:12:24	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-7q6zk to ip-10-0-141-83.us-east-2.compute.internal
22:55:38	kuadrant-system		dns-operator-controller-manager-8557cbdb84-xgv72	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgv72 to ip-10-0-138-89.us-east-2.compute.internal
23:11:46	kuadrant-system		authorino-6469b954d7-sd82d	Scheduled	Successfully assigned kuadrant-system/authorino-6469b954d7-sd82d to ip-10-0-128-124.us-east-2.compute.internal
23:15:32	opendatahub		test-curl-12475-4c2467	Scheduled	Successfully assigned opendatahub/test-curl-12475-4c2467 to ip-10-0-138-89.us-east-2.compute.internal
23:12:09	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-55s9l to ip-10-0-128-124.us-east-2.compute.internal
23:11:54	openshift-ingress		maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-4wxx9 to ip-10-0-128-124.us-east-2.compute.internal
23:13:56	llm		grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Scheduled	Successfully assigned llm/grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf to ip-10-0-141-83.us-east-2.compute.internal
22:55:47	openshift-ingress		kube-auth-proxy-685596c46f-qd744	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-685596c46f-qd744 to ip-10-0-141-83.us-east-2.compute.internal
22:55:14	opendatahub-operator-system		opendatahub-operator-controller-manager-8b65dd898-z95sw	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-8b65dd898-z95sw to ip-10-0-128-124.us-east-2.compute.internal
23:08:58	kuadrant-system		authorino-6889cbd8c4-ql6v4	Scheduled	Successfully assigned kuadrant-system/authorino-6889cbd8c4-ql6v4 to ip-10-0-128-124.us-east-2.compute.internal
23:14:16	opendatahub		test-curl-12475-02a120	Scheduled	Successfully assigned opendatahub/test-curl-12475-02a120 to ip-10-0-138-89.us-east-2.compute.internal
22:56:40	opendatahub		odh-model-controller-769875db56-7vgbb	Scheduled	Successfully assigned opendatahub/odh-model-controller-769875db56-7vgbb to ip-10-0-141-83.us-east-2.compute.internal
22:56:40	opendatahub		model-serving-api-75bd89fdfc-28xf7	Scheduled	Successfully assigned opendatahub/model-serving-api-75bd89fdfc-28xf7 to ip-10-0-141-83.us-east-2.compute.internal
23:08:09	opendatahub		maas-controller-857dc8657b-8kjqq	Scheduled	Successfully assigned opendatahub/maas-controller-857dc8657b-8kjqq to ip-10-0-141-83.us-east-2.compute.internal
23:08:09	opendatahub		maas-controller-7dd9588974-dr9xv	Scheduled	Successfully assigned opendatahub/maas-controller-7dd9588974-dr9xv to ip-10-0-141-83.us-east-2.compute.internal
22:56:40	opendatahub		llmisvc-controller-manager-869d5dcc5d-tw9c6	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-tw9c6 to ip-10-0-141-83.us-east-2.compute.internal
22:43:21	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-764475789f from 0 to 1
22:43:42	openshift-ingress	replicaset-controller	router-default-764475789f	SuccessfulCreate	Created pod: router-default-764475789f-szjgr
22:43:52	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
22:43:52	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
22:43:55	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
22:50:26 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
22:50:50 (x5)	openshift-ingress	kubelet	router-default-764475789f-szjgr	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
22:50:50 (x5)	openshift-ingress	kubelet	router-default-764475789f-szjgr	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
22:50:54	openshift-ingress	kubelet	router-default-764475789f-szjgr	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
22:50:55	openshift-ingress	kubelet	router-default-764475789f-szjgr	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
22:50:55	openshift-ingress	multus	router-default-764475789f-szjgr	AddedInterface	Add eth0 [10.132.0.11/23] from ovn-kubernetes
22:50:56	openshift-ingress	kubelet	router-default-764475789f-szjgr	Created	Container created
22:50:56	openshift-ingress	kubelet	router-default-764475789f-szjgr	Started	Container started
22:54:33	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
22:54:34	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
22:54:34	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-mmgdm
22:54:35	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
22:54:35	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-mmgdm	AddedInterface	Add eth0 [10.134.0.21/23] from ovn-kubernetes
22:54:52	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 17.612s (17.612s including waiting). Image size: 185476477 bytes.
22:54:52	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Started	Container started
22:54:52	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Created	Container created
22:54:53	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-864cbfbb6c from 0 to 1
22:54:53	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
22:54:53	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-864cbfbb6c	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b
22:54:53	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
22:54:53	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
22:54:53	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-68cb9b4b95 from 0 to 1
22:54:53	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-jtf6f
22:54:54	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
22:54:54	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
22:54:54	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
22:54:54	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
22:54:56	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
22:55:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.083s (8.083s including waiting). Image size: 183555333 bytes.
22:55:02	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	Started	Container started
22:55:02	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	Created	Container created
22:55:02	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-864cbfbb6c-n758b	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.531s (8.531s including waiting). Image size: 183555333 bytes.
22:55:12 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsNotMet	one or more requirements couldn't be found
22:55:12 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsUnknown	requirements not yet checked
22:55:14	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-8b65dd898	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-8b65dd898-w5dkz
22:55:14	opendatahub-operator-system	deployment-controller	opendatahub-operator-controller-manager	ScalingReplicaSet	Scaled up replica set opendatahub-operator-controller-manager-8b65dd898 from 0 to 3
22:55:14	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-8b65dd898	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-8b65dd898-z95sw
22:55:14	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-8b65dd898	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-8b65dd898-sf2v5
22:55:15	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-8b65dd898-w5dkz	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
22:55:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:15	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-8b65dd898-sf2v5	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
22:55:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:15	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-8b65dd898-z95sw	AddedInterface	Add eth0 [10.134.0.22/23] from ovn-kubernetes
22:55:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:16 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	AllRequirementsMet	all requirements found, attempting install
22:55:16	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	NeedsReinstall	calculated deployment install is bad
22:55:17	opendatahub-operator-system	operator-lifecycle-manager	install-jhx6t	AppliedWithWarnings	1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1
22:55:17	cert-manager	deployment-controller	cert-manager-webhook	ScalingReplicaSet	Scaled up replica set cert-manager-webhook-5dbbbf75cc from 0 to 1
22:55:17	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	SuccessfulCreate	Created pod: cert-manager-webhook-5dbbbf75cc-qssgz
22:55:17 (x4)	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	FailedCreate	Error creating: pods "cert-manager-webhook-5dbbbf75cc-" is forbidden: error looking up service account cert-manager/cert-manager-webhook: serviceaccount "cert-manager-webhook" not found
22:55:17 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	waiting for install components to report healthy
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Created	Container created
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Started	Container started
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Started	Container started
22:55:18 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability.
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 3.135s (3.135s including waiting). Image size: 273709574 bytes.
22:55:18	cert-manager	deployment-controller	cert-manager-cainjector	ScalingReplicaSet	Scaled up replica set cert-manager-cainjector-5c57bbd4fc from 0 to 1
22:55:18	cert-manager	multus	cert-manager-webhook-5dbbbf75cc-qssgz	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 3.599s (3.599s including waiting). Image size: 273709574 bytes.
22:55:18	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-qssgz	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Created	Container created
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 3.199s (3.199s including waiting). Image size: 273709574 bytes.
22:55:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Created	Container created
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 398ms (398ms including waiting). Image size: 273709574 bytes.
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:19 (x9)	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	FailedCreate	Error creating: pods "cert-manager-cainjector-5c57bbd4fc-" is forbidden: error looking up service account cert-manager/cert-manager-cainjector: serviceaccount "cert-manager-cainjector" not found
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Created	Container created
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-sf2v5	Started	Container started
22:55:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Started	Container started
22:55:19	opendatahub-operator-system	opendatahub-operator-controller-manager-8b65dd898-sf2v5_e8641dc2-b6f1-4534-956d-3f2128042a18	07ed84f7.opendatahub.io	LeaderElection	opendatahub-operator-controller-manager-8b65dd898-sf2v5_e8641dc2-b6f1-4534-956d-3f2128042a18 became leader
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 462ms (462ms including waiting). Image size: 273709574 bytes.
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 376ms (376ms including waiting). Image size: 273709574 bytes.
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Created	Container created
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Started	Container started
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-z95sw	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Started	Container started
22:55:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-8b65dd898-w5dkz	Created	Container created
22:55:21 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsNotMet	one or more requirements couldn't be found
22:55:21 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsUnknown	requirements not yet checked
22:55:21	cert-manager	multus	cert-manager-cainjector-5c57bbd4fc-59v6p	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
22:55:21	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-59v6p	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
22:55:21	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	SuccessfulCreate	Created pod: cert-manager-cainjector-5c57bbd4fc-59v6p
22:55:22	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-qssgz	Started	Container started
22:55:22	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-qssgz	Created	Container created
22:55:22	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-qssgz	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 3.991s (3.991s including waiting). Image size: 309650901 bytes.
22:55:23 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:55:23 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	waiting for install components to report healthy
22:55:23	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	AllRequirementsMet	all requirements found, attempting install
22:55:23	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
22:55:23	openshift-jobset-operator	replicaset-controller	jobset-operator-674cd59dc5	SuccessfulCreate	Created pod: jobset-operator-674cd59dc5-5s8w9
22:55:23	openshift-jobset-operator	deployment-controller	jobset-operator	ScalingReplicaSet	Scaled up replica set jobset-operator-674cd59dc5 from 0 to 1
22:55:23 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:55:23	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
22:55:24 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallWaiting	installing: waiting for deployment jobset-operator to become ready: deployment "jobset-operator" not available: Deployment does not have minimum availability.
22:55:24	openshift-jobset-operator	multus	jobset-operator-674cd59dc5-5s8w9	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
22:55:24	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-5s8w9	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e"
22:55:25	cert-manager	deployment-controller	cert-manager	ScalingReplicaSet	Scaled up replica set cert-manager-597b95cb84 from 0 to 1
22:55:26 (x9)	cert-manager	replicaset-controller	cert-manager-597b95cb84	FailedCreate	Error creating: pods "cert-manager-597b95cb84-" is forbidden: error looking up service account cert-manager/cert-manager: serviceaccount "cert-manager" not found
22:55:27	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-5s8w9	Started	Container started
22:55:27	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-5s8w9	Created	Container created
22:55:27	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-5s8w9	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e" in 2.892s (2.892s including waiting). Image size: 224874686 bytes.
22:55:27	openshift-jobset-operator	openshift-jobset-operator	openshift-jobset-operator-lock	LeaderElection	jobset-operator-674cd59dc5-5s8w9_e430ad98-f0db-49a8-8a16-d4c4f71ddab8 became leader
22:55:27	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	install strategy completed with no errors
22:55:28	cert-manager	kubelet	cert-manager-597b95cb84-m9rwx	Started	Container started
22:55:28	cert-manager	replicaset-controller	cert-manager-597b95cb84	SuccessfulCreate	Created pod: cert-manager-597b95cb84-m9rwx
22:55:28	cert-manager	kubelet	cert-manager-597b95cb84-m9rwx	Created	Container created
22:55:28	cert-manager	kubelet	cert-manager-597b95cb84-m9rwx	Pulled	Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod
22:55:28	cert-manager	multus	cert-manager-597b95cb84-m9rwx	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
22:55:31	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-678d87766f from 0 to 1
22:55:31	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-wc9rp
22:55:31	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
22:55:31	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-wc9rp	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
22:55:31	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
22:55:32	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-wc9rp	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"
22:55:33	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	install strategy completed with no errors
22:55:33	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-59v6p	Started	Container started
22:55:33	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-59v6p	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 12.244s (12.244s including waiting). Image size: 309650901 bytes.
22:55:33	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-59v6p	Created	Container created
22:55:34	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
22:55:35	kuadrant-system	multus	authorino-operator-7b7bb4bc77-fbm7v	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
22:55:35	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-7b7bb4bc77 from 0 to 1
22:55:35	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-fbm7v	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589"
22:55:35	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
22:55:35	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
22:55:35	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
22:55:35	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
22:55:35	kuadrant-system	replicaset-controller	authorino-operator-7b7bb4bc77	SuccessfulCreate	Created pod: authorino-operator-7b7bb4bc77-fbm7v
22:55:36	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-wc9rp	Created	Container created
22:55:36	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-wc9rp_b61504bf-3e9d-4ff9-9803-c5b6d7cb16a6	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-wc9rp_b61504bf-3e9d-4ff9-9803-c5b6d7cb16a6 became leader
22:55:36	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-wc9rp	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 4.515s (4.515s including waiting). Image size: 424244516 bytes.
22:55:36	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-wc9rp	Started	Container started
22:55:37	kuadrant-system	replicaset-controller	kuadrant-console-plugin-66c4fb575d	SuccessfulCreate	Created pod: kuadrant-console-plugin-66c4fb575d-j5bdq
22:55:37	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-66c4fb575d from 0 to 1
22:55:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
22:55:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
22:55:38	kuadrant-system	replicaset-controller	dns-operator-controller-manager-8557cbdb84	SuccessfulCreate	Created pod: dns-operator-controller-manager-8557cbdb84-xgv72
22:55:38	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-xgv72	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a"
22:55:38	kuadrant-system	multus	dns-operator-controller-manager-8557cbdb84-xgv72	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
22:55:38	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:55:38	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
22:55:38	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:55:38	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
22:55:38	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
22:55:38	kuadrant-system	multus	kuadrant-console-plugin-66c4fb575d-j5bdq	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
22:55:38	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-j5bdq	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044"
22:55:38	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-8557cbdb84 from 0 to 1
22:55:41	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-5778bb4449	SuccessfulCreate	Created pod: limitador-operator-controller-manager-5778bb4449-xgxbq
22:55:41	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-fbm7v	Created	Container created
22:55:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
22:55:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
22:55:41	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-5778bb4449 from 0 to 1
22:55:41	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-fbm7v	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589" in 5.441s (5.442s including waiting). Image size: 167738465 bytes.
22:55:41	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
22:55:41	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-fbm7v	Started	Container started
22:55:41	kuadrant-system	authorino-operator-7b7bb4bc77-fbm7v_f3c96102-9f68-4b0a-bbb8-9756e9e1e3a9	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-7b7bb4bc77-fbm7v_f3c96102-9f68-4b0a-bbb8-9756e9e1e3a9 became leader
22:55:42	kuadrant-system	multus	limitador-operator-controller-manager-5778bb4449-xgxbq	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
22:55:42	kuadrant-system	dns-operator-controller-manager-8557cbdb84-xgv72_69dec5a0-626a-4c37-9bc1-72ddbb61ddba	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-8557cbdb84-xgv72_69dec5a0-626a-4c37-9bc1-72ddbb61ddba became leader
22:55:42	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-xgv72	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a" in 3.231s (3.231s including waiting). Image size: 190804149 bytes.
22:55:42	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-xgv72	Created	Container created
22:55:42	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-xgv72	Started	Container started
22:55:42	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-xgxbq	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e"
22:55:44	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-xgxbq	Started	Container started
22:55:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-j5bdq	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044" in 6.424s (6.424s including waiting). Image size: 342538866 bytes.
22:55:44	kuadrant-system	limitador-operator-controller-manager-5778bb4449-xgxbq_6315a165-9306-40b6-b0a1-323e331f02d9	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-5778bb4449-xgxbq_6315a165-9306-40b6-b0a1-323e331f02d9 became leader
22:55:44	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-xgxbq	Created	Container created
22:55:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-j5bdq	Started	Container started
22:55:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-j5bdq	Created	Container created
22:55:44	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-xgxbq	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e" in 2.035s (2.035s including waiting). Image size: 177967631 bytes.
22:55:47	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-7d4ffc574f	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2
22:55:47	openshift-ingress	multus	kube-auth-proxy-685596c46f-qd744	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
22:55:47	openshift-ingress	replicaset-controller	kube-auth-proxy-685596c46f	SuccessfulCreate	Created pod: kube-auth-proxy-685596c46f-cvm95
22:55:47	openshift-ingress	replicaset-controller	kube-auth-proxy-685596c46f	SuccessfulCreate	Created pod: kube-auth-proxy-685596c46f-qd744
22:55:47	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-685596c46f from 0 to 2
22:55:47	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-7d4ffc574f from 0 to 1
22:55:47	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceCreated	Created Service/jobset-webhook-service -n openshift-jobset-operator because it was missing
22:55:47	openshift-ingress	multus	kube-auth-proxy-685596c46f-cvm95	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
22:55:47	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-cvm95	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
22:55:47	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-qd744	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceAccountCreated	Created ServiceAccount/jobset-controller-manager -n openshift-jobset-operator because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-manager-rolebinding because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-proxy-role because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-metrics-reader because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-manager-role because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-metrics-cert -n openshift-jobset-operator because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-serving-cert -n openshift-jobset-operator because it was missing
22:55:47	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	IssuerCreated	Created Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator because it was missing
22:55:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-manager-secrets-role -n openshift-jobset-operator because it was missing
22:55:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
22:55:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2	Created	Container created
22:55:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2	Started	Container started
22:55:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding -n openshift-jobset-operator because it was missing
22:55:48	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-7d4ffc574fm2gs2	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
22:55:48	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-leader-election-role -n openshift-jobset-operator because it was missing
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	opendatahub	replicaset-controller	kserve-module-controller-manager-7566cf5fc8	SuccessfulCreate	Created pod: kserve-module-controller-manager-7566cf5fc8-w29fq
22:55:49	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceCreated	Created Service/jobset-controller-manager-metrics-service -n openshift-jobset-operator because it was missing
22:55:49	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-metrics-cert	Issuing	The certificate has been successfully issued
22:55:49	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-metrics-cert	Requested	Created new CertificateRequest resource "jobset-metrics-cert-1"
22:55:49	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-metrics-cert	Generated	Stored new private key in temporary Secret resource "jobset-metrics-cert-85rb5"
22:55:49	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-metrics-cert	Issuing	Issuing certificate as Secret does not exist
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	CertificateIssued	Certificate fetched from issuer successfully
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-metrics-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
22:55:49	opendatahub	deployment-controller	kserve-module-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-manager-secrets-rolebinding -n openshift-jobset-operator because it was missing
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-serving-cert	Issuing	Issuing certificate as Secret does not exist
22:55:49	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-serving-cert	Generated	Stored new private key in temporary Secret resource "jobset-serving-cert-grsgn"
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-serving-cert	Requested	Created new CertificateRequest resource "jobset-serving-cert-1"
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-serving-cert	Issuing	The certificate has been successfully issued
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-serving-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
22:55:49	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	CertificateIssued	Certificate fetched from issuer successfully
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-qd744	Created	Container created
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-cvm95	Started	Container started
22:55:50	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulCreate	Created pod: authorino-57f7f8659f-rz2w9
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-cvm95	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.563s (2.563s including waiting). Image size: 188904677 bytes.
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-cvm95	Created	Container created
22:55:50	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationCreated	Created ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it was missing
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-qd744	Started	Container started
22:55:50	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationCreated	Created MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it was missing
22:55:50	openshift-ingress	kubelet	kube-auth-proxy-685596c46f-qd744	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.641s (2.641s including waiting). Image size: 188904677 bytes.
22:55:50	opendatahub	multus	kserve-module-controller-manager-7566cf5fc8-w29fq	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
22:55:50	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-w29fq	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189"
22:55:50	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-57f7f8659f from 0 to 1
22:55:50	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CustomResourceDefinitionCreated	Created CustomResourceDefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io because it was missing
22:55:51	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-w29fq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 1.509s (1.509s including waiting). Image size: 195729000 bytes.
22:55:51	openshift-jobset-operator	deployment-controller	jobset-controller-manager	ScalingReplicaSet	Scaled up replica set jobset-controller-manager-7c646f4476 from 0 to 1
22:55:51	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ConfigMapCreated	Created ConfigMap/jobset-manager-config -n openshift-jobset-operator because it was missing
22:55:51	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	DeploymentCreated	Created Deployment.apps/jobset-controller-manager -n openshift-jobset-operator because it was missing
22:55:51	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceMonitorCreated	Created ServiceMonitor.monitoring.coreos.com/jobset-controller-manager-metrics-monitor -n openshift-jobset-operator because it was missing
22:55:51	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-w29fq	Started	Container started
22:55:51	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
22:55:51	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
22:55:51	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-w29fq	Created	Container created
22:55:51	kuadrant-system	kubelet	authorino-57f7f8659f-rz2w9	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
22:55:51	kuadrant-system	multus	authorino-57f7f8659f-rz2w9	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
22:55:51	openshift-jobset-operator	replicaset-controller	jobset-controller-manager-7c646f4476	SuccessfulCreate	Created pod: jobset-controller-manager-7c646f4476-xp7tk
22:55:52	openshift-jobset-operator	multus	jobset-controller-manager-7c646f4476-xp7tk	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
22:55:52	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationUpdated	Updated MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it changed
22:55:52	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationUpdated	Updated ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it changed
22:55:52	openshift-jobset-operator	kubelet	jobset-controller-manager-7c646f4476-xp7tk	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987"
22:55:53	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
22:55:53	kuadrant-system	kubelet	authorino-57f7f8659f-rz2w9	Created	Container created
22:55:53	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
22:55:53	kuadrant-system	kubelet	authorino-57f7f8659f-rz2w9	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.418s (2.418s including waiting). Image size: 204096810 bytes.
22:55:53	kuadrant-system	kubelet	authorino-57f7f8659f-rz2w9	Started	Container started
22:55:56	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
22:55:56	openshift-jobset-operator	kubelet	jobset-controller-manager-7c646f4476-xp7tk	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" in 4.24s (4.24s including waiting). Image size: 182184678 bytes.
22:55:57	openshift-jobset-operator	jobset-controller-manager-7c646f4476-xp7tk_76b7a422-7107-4624-847b-f52850e042c1	6d4f6a47.jobset.x-k8s.io	LeaderElection	jobset-controller-manager-7c646f4476-xp7tk_76b7a422-7107-4624-847b-f52850e042c1 became leader
22:56:07	opendatahub	kserve-module-controller-manager-7566cf5fc8-w29fq_dff61cbe-a14b-4029-92f8-fb862437daa8	kserve-module-controller-manager	LeaderElection	kserve-module-controller-manager-7566cf5fc8-w29fq_dff61cbe-a14b-4029-92f8-fb862437daa8 became leader
22:56:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:19 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:19 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:40	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1
22:56:40	opendatahub	replicaset-controller	model-serving-api-75bd89fdfc	SuccessfulCreate	Created pod: model-serving-api-75bd89fdfc-28xf7
22:56:40	opendatahub	deployment-controller	model-serving-api	ScalingReplicaSet	Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1
22:56:40	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
22:56:40	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-hww68
22:56:40	opendatahub	replicaset-controller	kserve-controller-manager-776cc66d47	SuccessfulCreate	Created pod: kserve-controller-manager-776cc66d47-dfkvs
22:56:40	opendatahub	deployment-controller	kserve-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1
22:56:40	opendatahub	deployment-controller	odh-model-controller	ScalingReplicaSet	Scaled up replica set odh-model-controller-769875db56 from 0 to 1
22:56:40	opendatahub	replicaset-controller	odh-model-controller-769875db56	SuccessfulCreate	Created pod: odh-model-controller-769875db56-7vgbb
22:56:40	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulCreate	Created pod: llmisvc-controller-manager-869d5dcc5d-tw9c6
22:56:41	opendatahub	multus	llmisvc-controller-manager-869d5dcc5d-tw9c6	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
22:56:41	opendatahub	multus	model-serving-api-75bd89fdfc-28xf7	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
22:56:41	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-hww68	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
22:56:41	opendatahub	multus	odh-model-controller-769875db56-7vgbb	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
22:56:41	opendatahub	kubelet	odh-model-controller-769875db56-7vgbb	Pulling	Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775"
22:56:41	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-hww68	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
22:56:41	opendatahub	kubelet	kserve-controller-manager-776cc66d47-dfkvs	Pulling	Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d"
22:56:41	opendatahub	multus	kserve-controller-manager-776cc66d47-dfkvs	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
22:56:41	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-tw9c6	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
22:56:41	opendatahub	kubelet	model-serving-api-75bd89fdfc-28xf7	Pulling	Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8"
22:56:44	opendatahub	kubelet	model-serving-api-75bd89fdfc-28xf7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 3s (3s including waiting). Image size: 184925761 bytes.
22:56:44	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-tw9c6	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.113s (3.113s including waiting). Image size: 235100981 bytes.
22:56:44	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-tw9c6	Started	Container started
22:56:44	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-tw9c6	Created	Container created
22:56:44	opendatahub	kubelet	kserve-controller-manager-776cc66d47-dfkvs	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.321s (3.321s including waiting). Image size: 247676581 bytes.
22:56:44	opendatahub	kubelet	model-serving-api-75bd89fdfc-28xf7	Started	Container started
22:56:44	opendatahub	kubelet	model-serving-api-75bd89fdfc-28xf7	Created	Container created
22:56:44	opendatahub	kubelet	kserve-controller-manager-776cc66d47-dfkvs	Started	Container started
22:56:44	opendatahub	kubelet	kserve-controller-manager-776cc66d47-dfkvs	Created	Container created
22:56:45	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-hww68	Started	Container started
22:56:45	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-hww68	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.851s (3.851s including waiting). Image size: 185476477 bytes.
22:56:45	opendatahub	kubelet	odh-model-controller-769875db56-7vgbb	Created	Container created
22:56:45	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-hww68	Created	Container created
22:56:45	opendatahub	kubelet	odh-model-controller-769875db56-7vgbb	Started	Container started
22:56:45	opendatahub	kserve-controller-manager-776cc66d47-dfkvs_034e492a-2a6f-4949-8d0b-e3c23391d15a	kserve-controller-manager-leader-lock	LeaderElection	kserve-controller-manager-776cc66d47-dfkvs_034e492a-2a6f-4949-8d0b-e3c23391d15a became leader
22:56:45	opendatahub	kubelet	odh-model-controller-769875db56-7vgbb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.789s (3.789s including waiting). Image size: 236755567 bytes.
22:56:45	opendatahub	odh-model-controller-769875db56-7vgbb_21b88bf8-ab22-443a-ade7-05fe4366431b	odh-model-controller.opendatahub.io	LeaderElection	odh-model-controller-769875db56-7vgbb_21b88bf8-ab22-443a-ade7-05fe4366431b became leader
22:56:46	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-mmgdm
22:56:46	opendatahub	llmisvc-controller-manager-869d5dcc5d-tw9c6_b6786d41-a41d-4493-8812-00cc78f9e129	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-869d5dcc5d-tw9c6_b6786d41-a41d-4493-8812-00cc78f9e129 became leader
22:56:46	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
22:56:46	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-mmgdm	Killing	Stopping container discovery
22:56:49 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:49 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:57:04	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:57:04	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:06:06	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-rgdzs
23:06:06 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
23:06:06	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-wc9rp	Killing	Stopping container manager
23:06:06 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
23:06:06	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	NeedsReinstall	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
23:06:06	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	ComponentUnhealthy	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
23:06:07	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-rgdzs	Created	Container created
23:06:07 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
23:06:07	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-rgdzs	Pulled	Container image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" already present on machine and can be accessed by the pod
23:06:07	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-rgdzs	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
23:06:07	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-rgdzs	Started	Container started
23:06:18 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
23:06:25	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-rgdzs_1cac314c-fe73-46cf-9e1d-9e197b32251f	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-rgdzs_1cac314c-fe73-46cf-9e1d-9e197b32251f became leader
23:06:25	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-74dfcfb894 from 0 to 1
23:06:25	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulCreate	Created pod: authorino-56c6c9ddd5-cps9r
23:06:25	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-56c6c9ddd5 from 0 to 1
23:06:25	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulCreate	Created pod: limitador-limitador-74dfcfb894-dnwfl
23:06:26	kuadrant-system	multus	limitador-limitador-74dfcfb894-dnwfl	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
23:06:26	kuadrant-system	multus	authorino-56c6c9ddd5-cps9r	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
23:06:26	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-7c9b87d574 from 0 to 1
23:06:26	kuadrant-system	replicaset-controller	limitador-limitador-7c9b87d574	SuccessfulCreate	Created pod: limitador-limitador-7c9b87d574-vf2qv
23:06:26	kuadrant-system	kubelet	authorino-56c6c9ddd5-cps9r	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
23:06:26	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-dnwfl	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
23:06:26	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-vf2qv	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
23:06:26	kuadrant-system	multus	limitador-limitador-7c9b87d574-vf2qv	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
23:06:27	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-57f7f8659f from 1 to 0
23:06:27	kuadrant-system	kubelet	authorino-56c6c9ddd5-cps9r	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 943ms (943ms including waiting). Image size: 204096810 bytes.
23:06:27	kuadrant-system	kubelet	authorino-56c6c9ddd5-cps9r	Created	Container created
23:06:27	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulDelete	Deleted pod: authorino-57f7f8659f-rz2w9
23:06:27	kuadrant-system	kubelet	authorino-56c6c9ddd5-cps9r	Started	Container started
23:06:27	kuadrant-system	kubelet	authorino-57f7f8659f-rz2w9	Killing	Stopping container authorino
23:06:28	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-dnwfl	Created	Container created
23:06:28	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-dnwfl	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 1.606s (1.607s including waiting). Image size: 133379307 bytes.
23:06:28	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-vf2qv	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 1.394s (1.394s including waiting). Image size: 133379307 bytes.
23:06:28	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-dnwfl	Started	Container started
23:06:28	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-vf2qv	Created	Container created
23:06:28	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-vf2qv	Started	Container started
23:06:40	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulDelete	Deleted pod: limitador-limitador-74dfcfb894-dnwfl
23:06:40	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-74dfcfb894 from 1 to 0
23:06:40	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-dnwfl	Killing	Stopping container limitador
23:08:09	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-7dd9588974 from 0 to 1
23:08:09	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-857dc8657b from 0 to 1
23:08:09	opendatahub	replicaset-controller	maas-controller-7dd9588974	SuccessfulCreate	Created pod: maas-controller-7dd9588974-dr9xv
23:08:09	opendatahub	replicaset-controller	maas-controller-857dc8657b	SuccessfulCreate	Created pod: maas-controller-857dc8657b-8kjqq
23:08:09	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	FailedMount	MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found
23:08:10	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:4c61ff034a1be4ad4c6eb72a9b2c52e4447f7642c6323ca72ebf161024ae8758"
23:08:10	opendatahub	multus	maas-controller-7dd9588974-dr9xv	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
23:08:10	opendatahub	kubelet	maas-controller-7dd9588974-dr9xv	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:4c61ff034a1be4ad4c6eb72a9b2c52e4447f7642c6323ca72ebf161024ae8758"
23:08:10	opendatahub	multus	maas-controller-857dc8657b-8kjqq	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
23:08:13	opendatahub	kubelet	maas-controller-7dd9588974-dr9xv	Started	Container started
23:08:13	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	Created	Container created
23:08:13	opendatahub	kubelet	maas-controller-7dd9588974-dr9xv	Created	Container created
23:08:13	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	Started	Container started
23:08:13	opendatahub	kubelet	maas-controller-7dd9588974-dr9xv	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:4c61ff034a1be4ad4c6eb72a9b2c52e4447f7642c6323ca72ebf161024ae8758" in 2.864s (2.864s including waiting). Image size: 200094743 bytes.
23:08:13	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:4c61ff034a1be4ad4c6eb72a9b2c52e4447f7642c6323ca72ebf161024ae8758" in 2.812s (2.812s including waiting). Image size: 200094743 bytes.
23:08:14	opendatahub	deployment-controller	usage-logs-tenancy-proxy	ScalingReplicaSet	Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1
23:08:14	opendatahub	maas-controller-7dd9588974-dr9xv_a7747789-82bc-4cfc-8f8c-175b06958a53	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-7dd9588974-dr9xv_a7747789-82bc-4cfc-8f8c-175b06958a53 became leader
23:08:15	opendatahub	replicaset-controller	usage-logs-tenancy-proxy-9c5d49bcd	SuccessfulCreate	Created pod: usage-logs-tenancy-proxy-9c5d49bcd-2tgkz
23:08:15	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	Pulling	Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172"
23:08:15	opendatahub	multus	usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
23:08:24 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Started	Container started
23:08:24	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:08:24 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Created	Container created
23:08:25	opendatahub	kubelet	maas-controller-857dc8657b-8kjqq	Killing	Stopping container manager
23:08:25	opendatahub	replicaset-controller	maas-controller-857dc8657b	SuccessfulDelete	Deleted pod: maas-controller-857dc8657b-8kjqq
23:08:25	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled down replica set maas-controller-857dc8657b from 1 to 0
23:08:27	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	Started	Container started
23:08:27	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	Created	Container created
23:08:27	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-2tgkz	Pulled	Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 12.195s (12.195s including waiting). Image size: 1089502687 bytes.
23:08:29 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	ProbeError	Startup probe error: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused body:
23:08:29 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Unhealthy	Startup probe failed: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused
23:08:54 (x5)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:08:54 (x5)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:08:58	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulDelete	Deleted pod: authorino-6889cbd8c4-ql6v4
23:08:58	kuadrant-system	kubelet	authorino-6889cbd8c4-ql6v4	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
23:08:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6dd76875b5 from 0 to 1
23:08:58	kuadrant-system	replicaset-controller	authorino-6dd76875b5	SuccessfulDelete	Deleted pod: authorino-6dd76875b5-jghnb
23:08:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6dd76875b5 from 1 to 0
23:08:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6889cbd8c4 from 0 to 1
23:08:58	kuadrant-system	replicaset-controller	authorino-64545899bb	SuccessfulCreate	Created pod: authorino-64545899bb-nsg72
23:08:58	kuadrant-system	multus	authorino-6889cbd8c4-ql6v4	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
23:08:58	kuadrant-system	replicaset-controller	authorino-6dd76875b5	SuccessfulCreate	Created pod: authorino-6dd76875b5-jghnb
23:08:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6889cbd8c4 from 1 to 0
23:08:58	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulCreate	Created pod: authorino-6889cbd8c4-ql6v4
23:08:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-64545899bb from 0 to 1
23:08:59	kuadrant-system	multus	authorino-64545899bb-nsg72	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
23:08:59	kuadrant-system	kubelet	authorino-64545899bb-nsg72	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
23:08:59	kuadrant-system	kubelet	authorino-6889cbd8c4-ql6v4	Started	Container started
23:08:59	kuadrant-system	kubelet	authorino-6889cbd8c4-ql6v4	Created	Container created
23:08:59	kuadrant-system	kubelet	authorino-6889cbd8c4-ql6v4	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 950ms (950ms including waiting). Image size: 204096810 bytes.
23:08:59	kuadrant-system	kubelet	authorino-64545899bb-nsg72	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 661ms (661ms including waiting). Image size: 204096810 bytes.
23:08:59	kuadrant-system	kubelet	authorino-64545899bb-nsg72	Created	Container created
23:08:59	kuadrant-system	kubelet	authorino-64545899bb-nsg72	Started	Container started
23:09:00	kuadrant-system	kubelet	authorino-6889cbd8c4-ql6v4	Killing	Stopping container authorino
23:09:00	kuadrant-system	kubelet	authorino-56c6c9ddd5-cps9r	Killing	Stopping container authorino
23:09:00	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulDelete	Deleted pod: authorino-56c6c9ddd5-cps9r
23:09:00	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-56c6c9ddd5 from 1 to 0
23:09:43	kuadrant-system	multus	authorino-5cd48575b8-lbldd	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
23:09:43	kuadrant-system	replicaset-controller	authorino-5cd48575b8	SuccessfulCreate	Created pod: authorino-5cd48575b8-lbldd
23:09:43	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-5cd48575b8 from 0 to 1
23:09:44	kuadrant-system	kubelet	authorino-5cd48575b8-lbldd	Started	Container started
23:09:44	kuadrant-system	kubelet	authorino-5cd48575b8-lbldd	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
23:09:44	kuadrant-system	kubelet	authorino-5cd48575b8-lbldd	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 903ms (903ms including waiting). Image size: 204096810 bytes.
23:09:44	kuadrant-system	kubelet	authorino-5cd48575b8-lbldd	Created	Container created
23:09:45	kuadrant-system	replicaset-controller	authorino-64545899bb	SuccessfulDelete	Deleted pod: authorino-64545899bb-nsg72
23:09:45	kuadrant-system	kubelet	authorino-64545899bb-nsg72	Killing	Stopping container authorino
23:09:46	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1
23:09:46	opendatahub	replicaset-controller	llmisvc-controller-manager-6466c554d8	SuccessfulCreate	Created pod: llmisvc-controller-manager-6466c554d8-fl2c7
23:09:47	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-fl2c7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 526ms (526ms including waiting). Image size: 235100981 bytes.
23:09:47	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-fl2c7	Created	Container created
23:09:47	opendatahub	multus	llmisvc-controller-manager-6466c554d8-fl2c7	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
23:09:47	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-fl2c7	Started	Container started
23:09:47	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-fl2c7	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
23:10:19	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0
23:10:19	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulDelete	Deleted pod: llmisvc-controller-manager-869d5dcc5d-tw9c6
23:10:19	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-tw9c6	Killing	Stopping container manager
23:10:22	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-pnbq9	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
23:10:22	opendatahub	multus	ai-gateway-controller-5d8dddb676-pnbq9	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
23:10:22	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulCreate	Created pod: ai-gateway-controller-5d8dddb676-pnbq9
23:10:22	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-5d8dddb676 from 0 to 1
23:10:26	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-pnbq9	Started	Container started
23:10:26	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-pnbq9	Created	Container created
23:10:26	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-pnbq9	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 3.876s (3.876s including waiting). Image size: 184136103 bytes.
23:10:27	opendatahub	ai-gateway-controller-5d8dddb676-pnbq9_7ffadd46-07a5-4507-a0c7-75513102dd64	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-5d8dddb676-pnbq9_7ffadd46-07a5-4507-a0c7-75513102dd64 became leader
23:10:28	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-mrg4r
23:10:28	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-7bb9489676 from 0 to 1
23:10:28	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	SuccessfulCreate	Created pod: payload-processing-7bb9489676-59ml8
23:10:28 (x4)	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	FailedCreate	Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
23:10:28 (x3)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
23:10:28	openshift-ingress	multus	payload-pre-processing-7b98bb678c-mrg4r	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
23:10:28	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
23:10:29	openshift-ingress	multus	payload-processing-7bb9489676-59ml8	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
23:10:29	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-mrg4r	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
23:10:29	openshift-ingress	kubelet	payload-processing-7bb9489676-59ml8	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
23:10:31	openshift-ingress	kubelet	payload-processing-7bb9489676-59ml8	Started	Container started
23:10:31	openshift-ingress	kubelet	payload-processing-7bb9489676-59ml8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.112s (2.112s including waiting). Image size: 125353963 bytes.
23:10:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-mrg4r	Created	Container created
23:10:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-mrg4r	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.128s (2.128s including waiting). Image size: 125353963 bytes.
23:10:31	openshift-ingress	kubelet	payload-processing-7bb9489676-59ml8	Created	Container created
23:10:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-mrg4r	Started	Container started
23:10:38	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-95895fbfc from 0 to 1
23:10:38	opendatahub	replicaset-controller	ai-gateway-controller-95895fbfc	SuccessfulCreate	Created pod: ai-gateway-controller-95895fbfc-2qtm9
23:10:38	opendatahub	kubelet	ai-gateway-controller-95895fbfc-2qtm9	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
23:10:38	opendatahub	multus	ai-gateway-controller-95895fbfc-2qtm9	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
23:10:41	opendatahub	kubelet	ai-gateway-controller-95895fbfc-2qtm9	Started	Container started
23:10:41	opendatahub	llmisvc-controller-manager-6466c554d8-fl2c7_c7debd14-36a0-45db-a312-1a885bd5d96f	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-6466c554d8-fl2c7_c7debd14-36a0-45db-a312-1a885bd5d96f became leader
23:10:41	opendatahub	kubelet	ai-gateway-controller-95895fbfc-2qtm9	Created	Container created
23:10:41	opendatahub	kubelet	ai-gateway-controller-95895fbfc-2qtm9	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 2.279s (2.279s including waiting). Image size: 184136103 bytes.
23:10:52 (x2)	openshift-jobset-operator	kubelet	jobset-controller-manager-7c646f4476-xp7tk	Started	Container started
23:10:52	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-pnbq9	Killing	Stopping container manager
23:10:52 (x2)	openshift-jobset-operator	kubelet	jobset-controller-manager-7c646f4476-xp7tk	Created	Container created
23:10:52	openshift-jobset-operator	kubelet	jobset-controller-manager-7c646f4476-xp7tk	Pulled	Container image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" already present on machine and can be accessed by the pod
23:10:52	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled down replica set ai-gateway-controller-5d8dddb676 from 1 to 0
23:10:52	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulDelete	Deleted pod: ai-gateway-controller-5d8dddb676-pnbq9
23:11:09	openshift-jobset-operator	jobset-controller-manager-7c646f4476-xp7tk_6a2940fc-9063-43bd-9ddd-f33e50ca26c3	6d4f6a47.jobset.x-k8s.io	LeaderElection	jobset-controller-manager-7c646f4476-xp7tk_6a2940fc-9063-43bd-9ddd-f33e50ca26c3 became leader
23:11:09	opendatahub	ai-gateway-controller-95895fbfc-2qtm9_dc883939-7d38-423a-87e3-38a7ad32a85c	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-95895fbfc-2qtm9_dc883939-7d38-423a-87e3-38a7ad32a85c became leader
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-2-simulated-kserve-workload-svc
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-2-simulated-prometheus-scraping
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Deployment llm/e2e-distinct-2-simulated-kserve
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-2-simulated-kserve-route
23:11:19	llm	multus	e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine-default
23:11:19	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:19	llm	replicaset-controller	e2e-distinct-2-simulated-kserve-cbd4f5586	SuccessfulCreate	Created pod: e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt
23:11:19	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Service llm/e2e-distinct-2-simulated-kserve-workload-svc
23:11:19	llm	deployment-controller	e2e-distinct-2-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-2-simulated-kserve-cbd4f5586 from 0 to 1
23:11:21	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	Created	Container created
23:11:21	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Deployment llm/e2e-distinct-simulated-kserve
23:11:21	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	Started	Container started
23:11:21	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Service llm/e2e-distinct-simulated-kserve-workload-svc
23:11:21	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-cd4dt	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.531s (1.531s including waiting). Image size: 98346788 bytes.
23:11:21	llm	deployment-controller	e2e-distinct-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-simulated-kserve-7b998b79d9 from 0 to 1
23:11:21	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
23:11:22	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:22	llm	multus	e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
23:11:22	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-simulated-kserve-route
23:11:22	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-simulated-kserve-workload-svc
23:11:22	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine-default
23:11:22	llm	replicaset-controller	e2e-distinct-simulated-kserve-7b998b79d9	SuccessfulCreate	Created pod: e2e-distinct-simulated-kserve-7b998b79d9-6nbz5
23:11:22	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine
23:11:22	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-simulated-prometheus-scraping
23:11:23	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:23	llm	multus	e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.NetworkPolicy llm/e2e-embedding-simulated-prometheus-scraping
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Service llm/e2e-embedding-simulated-kserve-workload-svc
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Deployment llm/e2e-embedding-simulated-kserve
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
23:11:23	llm	deployment-controller	e2e-embedding-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-embedding-simulated-kserve-67bf594cd6 from 0 to 1
23:11:23	llm	replicaset-controller	e2e-embedding-simulated-kserve-67bf594cd6	SuccessfulCreate	Created pod: e2e-embedding-simulated-kserve-67bf594cd6-pkjh4
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.HTTPRoute llm/e2e-embedding-simulated-kserve-route
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine-default
23:11:23	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.DestinationRule llm/e2e-embedding-simulated-kserve-workload-svc
23:11:24	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	Created	Container created
23:11:24	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	Created	Container created
23:11:24	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 313ms (313ms including waiting). Image size: 98346788 bytes.
23:11:24	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-pkjh4	Started	Container started
23:11:24	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	Started	Container started
23:11:24	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-6nbz5	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.402s (1.402s including waiting). Image size: 98346788 bytes.
23:11:26	llm	deployment-controller	e2e-trlp-test-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-trlp-test-simulated-kserve-69b588897d from 0 to 1
23:11:26	llm	multus	e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
23:11:26	llm	replicaset-controller	e2e-trlp-test-simulated-kserve-69b588897d	SuccessfulCreate	Created pod: e2e-trlp-test-simulated-kserve-69b588897d-bcnjr
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Service llm/e2e-trlp-test-simulated-kserve-workload-svc
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.HTTPRoute llm/e2e-trlp-test-simulated-kserve-route
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.DestinationRule llm/e2e-trlp-test-simulated-kserve-workload-svc
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine-default
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.NetworkPolicy llm/e2e-trlp-test-simulated-prometheus-scraping
23:11:26	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Deployment llm/e2e-trlp-test-simulated-kserve
23:11:27	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	Started	Container started
23:11:27	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	Created	Container created
23:11:27	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 284ms (284ms including waiting). Image size: 98346788 bytes.
23:11:27	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-bcnjr	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/e2e-unconab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Deployment llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Service llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-route
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/eab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine-default
23:11:28	llm	deployment-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc from 0 to 1
23:11:28	llm	replicaset-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc	SuccessfulCreate	Created pod: e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6
23:11:28	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-prometheus-scraping
23:11:29	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 323ms (323ms including waiting). Image size: 98346788 bytes.
23:11:29	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	Created	Container created
23:11:29	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:29	llm	multus	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
23:11:29	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87gnr6	Started	Container started
23:11:29	llm	replicaset-controller	echo-simulator-kserve-64b6f8f96c	SuccessfulCreate	Created pod: echo-simulator-kserve-64b6f8f96c-6q4sf
23:11:29	llm	deployment-controller	echo-simulator-kserve	ScalingReplicaSet	Scaled up replica set echo-simulator-kserve-64b6f8f96c from 0 to 1
23:11:29	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Secret llm/echo-simulator-kserve-self-signed-certs
23:11:29	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Deployment llm/echo-simulator-kserve
23:11:29	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Service llm/echo-simulator-kserve-workload-svc
23:11:29	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.HTTPRoute llm/echo-simulator-kserve-route
23:11:29	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.DestinationRule llm/echo-simulator-kserve-workload-svc
23:11:30	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.NetworkPolicy llm/echo-simulator-prometheus-scraping
23:11:30	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine
23:11:30	llm	multus	echo-simulator-kserve-64b6f8f96c-6q4sf	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
23:11:30	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6q4sf	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0"
23:11:30	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine-default
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine
23:11:31	llm	replicaset-controller	facebook-opt-125m-simulated-kserve-84b4ff656c	SuccessfulCreate	Created pod: facebook-opt-125m-simulated-kserve-84b4ff656c-9d645
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/facebook-opt-125m-simulated-kserve-workload-svc
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine-default
23:11:31	llm	deployment-controller	facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set facebook-opt-125m-simulated-kserve-84b4ff656c from 0 to 1
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/facebook-opt-125m-simulated-prometheus-scraping
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Service llm/facebook-opt-125m-simulated-kserve-workload-svc
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
23:11:31	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:31	llm	multus	facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Deployment llm/facebook-opt-125m-simulated-kserve
23:11:31	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/facebook-opt-125m-simulated-kserve-route
23:11:32	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	Created	Container created
23:11:32	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6q4sf	Started	Container started
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Service llm/premium-simulated-simulated-premium-kserve-workload-svc
23:11:32	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	Started	Container started
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.HTTPRoute llm/premium-simulated-simulated-premium-kserve-route
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.DestinationRule llm/premium-simulated-simulated-premium-kserve-workload-svc
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Deployment llm/premium-simulated-simulated-premium-kserve
23:11:32	llm	replicaset-controller	premium-simulated-simulated-premium-kserve-5f5c4b459f	SuccessfulCreate	Created pod: premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/p75d89895ad19b4ac62612d08bfe7f574-kserve-llmisvc-engine-default
23:11:32	llm	deployment-controller	premium-simulated-simulated-premium-kserve	ScalingReplicaSet	Scaled up replica set premium-simulated-simulated-premium-kserve-5f5c4b459f from 0 to 1
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
23:11:32	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-9d645	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 281ms (281ms including waiting). Image size: 98346788 bytes.
23:11:32	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6q4sf	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0" in 1.846s (1.846s including waiting). Image size: 78208878 bytes.
23:11:32	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6q4sf	Created	Container created
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/premium-simulated-simulated-premium-kserve-llmisvc-engine
23:11:32	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.NetworkPolicy llm/premium-simulated-simulated-premium-prometheus-scraping
23:11:33	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 295ms (295ms including waiting). Image size: 98346788 bytes.
23:11:33	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-2-simulated] is Ready
23:11:33	llm	multus	premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
23:11:33 (x2)	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Updated	Updated v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
23:11:33	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	Created	Container created
23:11:33	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
23:11:33	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-w8fvn	Started	Container started
23:11:34	llm	LLMInferenceServiceController	e2e-distinct-simulated	Updated	Updated v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
23:11:34	llm	LLMInferenceServiceController	e2e-distinct-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-simulated] is Ready
23:11:35	llm	LLMInferenceServiceController	e2e-embedding-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-embedding-simulated] is Ready
23:11:35	llm	LLMInferenceServiceController	e2e-embedding-simulated	Updated	Updated v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
23:11:37	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Updated	Updated v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
23:11:38	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
23:11:38	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-trlp-test-simulated] is Ready
23:11:39	llm	LLMInferenceServiceController	echo-simulator	Updated	Updated v1.Secret llm/echo-simulator-kserve-self-signed-certs
23:11:41	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
23:11:41	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [facebook-opt-125m-simulated] is Ready
23:11:42	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Updated	Updated v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
23:11:43	llm	LLMInferenceServiceController	echo-simulator	LLMInferenceServiceReady	LLMInferenceService [echo-simulator] is Ready
23:11:43	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-unconfigured-facebook-opt-125m-simulated] is Ready
23:11:44	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	LLMInferenceServiceReady	LLMInferenceService [premium-simulated-simulated-premium] is Ready
23:11:46	kuadrant-system	kubelet	authorino-6469b954d7-sd82d	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
23:11:46	kuadrant-system	multus	authorino-6469b954d7-sd82d	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
23:11:46	kuadrant-system	replicaset-controller	authorino-6469b954d7	SuccessfulCreate	Created pod: authorino-6469b954d7-sd82d
23:11:54	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-kjjnb
23:11:54	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-4wxx9
23:11:54	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-68cb9b4b95 from 1 to 4
23:11:54	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
23:11:54	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-gxm54
23:11:54	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
23:11:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:11:55	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
23:11:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Created	Container created
23:11:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Started	Container started
23:11:55	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
23:11:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
23:11:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
23:11:58	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Started	Container started
23:11:58	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Created	Container created
23:11:58	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.971s (2.971s including waiting). Image size: 183555333 bytes.
23:11:59	kuadrant-system	kubelet	authorino-6469b954d7-sd82d	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 13.058s (13.058s including waiting). Image size: 204096810 bytes.
23:11:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.885s (4.885s including waiting). Image size: 183555333 bytes.
23:11:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Created	Container created
23:11:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Started	Container started
23:11:59	kuadrant-system	kubelet	authorino-6469b954d7-sd82d	Started	Container started
23:11:59	kuadrant-system	kubelet	authorino-6469b954d7-sd82d	Created	Container created
23:12:00	kuadrant-system	kubelet	authorino-5cd48575b8-lbldd	Killing	Stopping container authorino
23:12:00 (x3)	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	(combined from similar events): Scaled down replica set authorino-5cd48575b8 from 1 to 0
23:12:00	kuadrant-system	replicaset-controller	authorino-5cd48575b8	SuccessfulDelete	Deleted pod: authorino-5cd48575b8-lbldd
23:12:01 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	ProbeError	Startup probe error: Get "http://10.133.0.36:15021/healthz/ready": dial tcp 10.133.0.36:15021: connect: connection refused body:
23:12:01 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Unhealthy	Startup probe failed: Get "http://10.133.0.36:15021/healthz/ready": dial tcp 10.133.0.36:15021: connect: connection refused
23:12:04 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	ProbeError	Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused body:
23:12:04 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Unhealthy	Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused
23:12:06 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Unhealthy	Startup probe failed: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused
23:12:06 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	ProbeError	Startup probe error: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused body:
23:12:09	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-g9ntf
23:12:09	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-pf29j
23:12:09	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-gpnwl
23:12:09	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-55s9l
23:12:09	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-68cb9b4b95 from 4 to 8
23:12:09	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason: cpu resource utilization (percentage of request) above target
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Created	Container created
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:12:10	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Created	Container created
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Started	Container started
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Started	Container started
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Created	Container created
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:12:10	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Started	Container started
23:12:10	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Started	Container started
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:12:10	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
23:12:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Created	Container created
23:12:16 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	ProbeError	Startup probe error: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused body:
23:12:16 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Unhealthy	Startup probe failed: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused
23:12:16 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Unhealthy	Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused
23:12:16 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	ProbeError	Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused body:
23:12:17 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Unhealthy	Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused
23:12:17 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	ProbeError	Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused body:
23:12:17 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	ProbeError	Startup probe error: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused body:
23:12:17 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Unhealthy	Startup probe failed: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused
23:12:24	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-68cb9b4b95 from 8 to 9
23:12:24	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 9; reason:
23:12:24	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-7q6zk
23:12:25	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
23:12:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:12:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Created	Container created
23:12:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Started	Container started
23:12:31 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Unhealthy	Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused
23:12:31 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	ProbeError	Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
23:12:38	llm	deployment-controller	grid-inv-9698c1f0-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-9698c1f0-model-kserve-57859b89fb from 0 to 1
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.Service llm/grid-inv-9698c1f0-model-kserve-workload-svc
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.DestinationRule llm/grid-inv-9698c1f0-model-kserve-workload-svc
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.Secret llm/grid-inv-9698c1f0-model-kserve-self-signed-certs
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.PodMonitor llm/grid-inv-9698c1f0-model-kserve-llmisvc-engine
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.PodMonitor llm/grid-inv-9698c1f0-model-kserve-llmisvc-engine-default
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.Deployment llm/grid-inv-9698c1f0-model-kserve
23:12:38	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.HTTPRoute llm/grid-inv-9698c1f0-model-kserve-route
23:12:38	llm	replicaset-controller	grid-inv-9698c1f0-model-kserve-57859b89fb	SuccessfulCreate	Created pod: grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2
23:12:39	llm	kubelet	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Created	Container created
23:12:39	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Created	Created v1.NetworkPolicy llm/grid-inv-9698c1f0-model-prometheus-scraping
23:12:39	llm	kubelet	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Started	Container started
23:12:39	llm	multus	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
23:12:39	llm	kubelet	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
23:12:42	llm	deployment-controller	grid-inv-9698c1f0-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-9698c1f0-unmatched-kserve-8bc6778b from 0 to 1
23:12:42	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.DestinationRule llm/grid-inv-9698c1f0-unmatched-kserve-workload-svc
23:12:42	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.Deployment llm/grid-inv-9698c1f0-unmatched-kserve
23:12:42	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-9698c1f0-unmatched-kserve-route
23:12:42	llm	replicaset-controller	grid-inv-9698c1f0-unmatched-kserve-8bc6778b	SuccessfulCreate	Created pod: grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s
23:12:42	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.Secret llm/grid-inv-9698c1f0-unmatched-kserve-self-signed-certs
23:12:42	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.Service llm/grid-inv-9698c1f0-unmatched-kserve-workload-svc
23:12:43	llm	kubelet	grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	Created	Container created
23:12:43	llm	kubelet	grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
23:12:43	llm	multus	grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
23:12:43	llm	kubelet	grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	Started	Container started
23:12:43	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.PodMonitor llm/grid-inv-9698c1f0-unmatched-kserve-llmisvc-engine-default
23:12:43	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.PodMonitor llm/grid-inv-9698c1f0-unmatched-kserve-llmisvc-engine
23:12:43	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-9698c1f0-unmatched-prometheus-scraping
23:12:44 (x2)	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	Updated	Updated v1.Secret llm/grid-inv-9698c1f0-model-kserve-self-signed-certs
23:12:49 (x2)	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	Updated	Updated v1.Secret llm/grid-inv-9698c1f0-unmatched-kserve-self-signed-certs
23:12:50	llm	LLMInferenceServiceController	grid-inv-9698c1f0-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-9698c1f0-model] is Ready
23:12:53	llm	LLMInferenceServiceController	grid-inv-9698c1f0-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-9698c1f0-unmatched] is Ready
23:13:09	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-68cb9b4b95 from 9 to 10
23:13:09	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
23:13:09	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-68cb9b4b95-ddn95
23:13:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:13:10	openshift-ingress	multus	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
23:13:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Created	Container created
23:13:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Started	Container started
23:13:18 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	ProbeError	Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused body:
23:13:18 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Unhealthy	Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused
23:13:21 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	ProbeError	Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:13:21 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Unhealthy	Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:13:50	llm	kubelet	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Killing	Stopping container main
23:13:51	llm	kubelet	grid-inv-9698c1f0-unmatched-kserve-8bc6778b-ql48s	Killing	Stopping container main
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.PodMonitor llm/grid-inv-a485b453-model-kserve-llmisvc-engine
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.Deployment llm/grid-inv-a485b453-model-kserve
23:13:56	llm	replicaset-controller	grid-inv-a485b453-model-kserve-68cb9dd9c6	SuccessfulCreate	Created pod: grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf
23:13:56	llm	deployment-controller	grid-inv-a485b453-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-a485b453-model-kserve-68cb9dd9c6 from 0 to 1
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.Secret llm/grid-inv-a485b453-model-kserve-self-signed-certs
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.Service llm/grid-inv-a485b453-model-kserve-workload-svc
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.HTTPRoute llm/grid-inv-a485b453-model-kserve-route
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.DestinationRule llm/grid-inv-a485b453-model-kserve-workload-svc
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.NetworkPolicy llm/grid-inv-a485b453-model-prometheus-scraping
23:13:56	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Created	Created v1.PodMonitor llm/grid-inv-a485b453-model-kserve-llmisvc-engine-default
23:13:57	llm	multus	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
23:13:57	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1"
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.Secret llm/grid-inv-a485b453-unmatched-kserve-self-signed-certs
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-a485b453-unmatched-prometheus-scraping
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.PodMonitor llm/grid-inv-a485b453-unmatched-kserve-llmisvc-engine
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.PodMonitor llm/grid-inv-a485b453-unmatched-kserve-llmisvc-engine-default
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.DestinationRule llm/grid-inv-a485b453-unmatched-kserve-workload-svc
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.Service llm/grid-inv-a485b453-unmatched-kserve-workload-svc
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-a485b453-unmatched-kserve-route
23:13:58	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Created	Created v1.Deployment llm/grid-inv-a485b453-unmatched-kserve
23:13:58	llm	replicaset-controller	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd	SuccessfulCreate	Created pod: grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm
23:13:58	llm	deployment-controller	grid-inv-a485b453-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-a485b453-unmatched-kserve-57c6dfb7cd from 0 to 1
23:13:59	llm	kubelet	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	Created	Container created
23:13:59	llm	kubelet	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
23:13:59	llm	multus	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
23:13:59	llm	kubelet	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	Started	Container started
23:14:01 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Unhealthy	Readiness probe failed: Get "http://10.133.0.20:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:14:01 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	ProbeError	Readiness probe error: Get "http://10.133.0.20:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:14:02	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" in 5.525s (5.525s including waiting). Image size: 98346788 bytes.
23:14:03	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Started	Container started
23:14:03	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Created	Container created
23:14:05	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	Updated	Updated v1.Secret llm/grid-inv-a485b453-unmatched-kserve-self-signed-certs
23:14:05	llm	kubelet	grid-inv-9698c1f0-model-kserve-57859b89fb-mrpl2	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8000/ready": dial tcp 10.133.0.38:8000: connect: connection refused
23:14:07 (x2)	llm	LLMInferenceServiceController	grid-inv-a485b453-model	Updated	Updated v1.Secret llm/grid-inv-a485b453-model-kserve-self-signed-certs
23:14:08	llm	LLMInferenceServiceController	grid-inv-a485b453-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-a485b453-model] is Ready
23:14:09	llm	LLMInferenceServiceController	grid-inv-a485b453-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-a485b453-unmatched] is Ready
23:14:12	opendatahub	kubelet	test-curl-12475-f2203b	Pulling	Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"
23:14:12	opendatahub	multus	test-curl-12475-f2203b	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
23:14:15	opendatahub	kubelet	test-curl-12475-f2203b	Pulled	Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.568s (2.568s including waiting). Image size: 106100986 bytes.
23:14:15	opendatahub	kubelet	test-curl-12475-f2203b	Created	Container created
23:14:15	opendatahub	kubelet	test-curl-12475-f2203b	Started	Container started
23:14:17	opendatahub	kubelet	test-curl-12475-f2203b	Killing	Stopping container test-curl-12475-f2203b
23:14:17	opendatahub	kubelet	test-curl-12475-02a120	Started	Container started
23:14:17	opendatahub	kubelet	test-curl-12475-02a120	Created	Container created
23:14:17	opendatahub	kubelet	test-curl-12475-02a120	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
23:14:17	opendatahub	multus	test-curl-12475-02a120	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
23:14:19	opendatahub	kubelet	test-curl-12475-02a120	Killing	Stopping container test-curl-12475-02a120
23:14:19	opendatahub	kubelet	test-curl-12475-65b8a9	Started	Container started
23:14:19	opendatahub	kubelet	test-curl-12475-65b8a9	Created	Container created
23:14:19	opendatahub	kubelet	test-curl-12475-65b8a9	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
23:14:19	opendatahub	multus	test-curl-12475-65b8a9	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
23:14:21	openshift-ingress	deployment-controller	e2e-shared-a-main-70a7bf-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df from 0 to 1
23:14:21	opendatahub	kubelet	test-curl-12475-65b8a9	Killing	Stopping container test-curl-12475-65b8a9
23:14:21	openshift-ingress	multus	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
23:14:21	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-70a7bf	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-70a7bf
23:14:21	openshift-ingress	replicaset-controller	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df	SuccessfulCreate	Created pod: e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n
23:14:26	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
23:14:26	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-70a7bf	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-70a7bf-555878474 from 0 to 1
23:14:26	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-70a7bf	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddc from 0 to 1
23:14:26	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddc	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs
23:14:26 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddc	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-70a7bf: serviceaccount "payload-processing-e2e-shared-a-main-70a7bf" not found
23:14:26 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-70a7bf-555878474	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-70a7bf-555878474-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-70a7bf: serviceaccount "payload-processing-e2e-shared-a-main-70a7bf" not found
23:14:26	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-70a7bf-555878474	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5
23:14:27	openshift-ingress	multus	payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
23:14:27	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:14:27	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	Created	Container created
23:14:27	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:14:27	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	Created	Container created
23:14:27	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	Started	Container started
23:14:27	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	Started	Container started
23:14:38	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Unhealthy	Readiness probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused
23:14:38	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	ProbeError	Readiness probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused body:
23:14:38 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:14:39 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Started	Container started
23:14:39 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Created	Container created
23:14:40	openshift-ingress	replicaset-controller	e2e-shared-b-main-8235df-openshift-default-7b66c97d45	SuccessfulCreate	Created pod: e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b
23:14:40	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-8235df	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-8235df
23:14:40	openshift-ingress	deployment-controller	e2e-shared-b-main-8235df-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-8235df-openshift-default-7b66c97d45 from 0 to 1
23:14:41	openshift-ingress	multus	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
23:14:44 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	ProbeError	Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused body:
23:14:44 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Unhealthy	Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused
23:14:45	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-8235df-565c44d86	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b
23:14:45	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-8235df	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-8235df-565c44d86 from 0 to 1
23:14:45 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-8235df: serviceaccount "payload-processing-e2e-shared-b-main-8235df" not found
23:14:45 (x4)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-8235df-565c44d86	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-8235df-565c44d86-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-8235df: serviceaccount "payload-processing-e2e-shared-b-main-8235df" not found
23:14:45	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2
23:14:45	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-8235df	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47 from 0 to 1
23:14:46	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
23:14:46	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	Created	Container created
23:14:46	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	Started	Container started
23:14:46	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	Started	Container started
23:14:46	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	Created	Container created
23:14:46	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:14:46	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
23:14:46	openshift-ingress	multus	payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
23:14:58	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	ProbeError	Readiness probe error: Get "http://10.133.0.53:15021/healthz/ready": read tcp 10.133.0.2:40616->10.133.0.53:15021: read: connection reset by peer body:
23:14:58	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Unhealthy	Readiness probe failed: Get "http://10.133.0.53:15021/healthz/ready": read tcp 10.133.0.2:40616->10.133.0.53:15021: read: connection reset by peer
23:14:59 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
23:14:59 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Created	Container created
23:14:59 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Started	Container started
23:15:04 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Unhealthy	Startup probe failed: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused
23:15:04 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	ProbeError	Startup probe error: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused body:
23:15:06 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-70a7bf-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:06 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-70a7bf-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:25 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-8235df-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:25 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-8235df-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:33	opendatahub	kubelet	test-curl-12475-4c2467	Created	Container created
23:15:33	opendatahub	multus	test-curl-12475-4c2467	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
23:15:33	opendatahub	kubelet	test-curl-12475-4c2467	Started	Container started
23:15:33	opendatahub	kubelet	test-curl-12475-4c2467	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
23:15:36	opendatahub	kubelet	test-curl-12475-4c2467	Killing	Stopping container test-curl-12475-4c2467
23:15:40	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-8235df-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:15:40	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-8235df-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:16:41	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Killing	Stopping container main
23:16:42	llm	kubelet	grid-inv-a485b453-unmatched-kserve-57c6dfb7cd-8vsqm	Killing	Stopping container main
23:16:46	llm	replicaset-controller	grid-inv-395bb97b-model-kserve-59fb55d785	SuccessfulCreate	Created pod: grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk
23:16:46	llm	deployment-controller	grid-inv-395bb97b-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-395bb97b-model-kserve-59fb55d785 from 0 to 1
23:16:46	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.Secret llm/grid-inv-395bb97b-model-kserve-self-signed-certs
23:16:46	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.Deployment llm/grid-inv-395bb97b-model-kserve
23:16:46	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.Service llm/grid-inv-395bb97b-model-kserve-workload-svc
23:16:46	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.DestinationRule llm/grid-inv-395bb97b-model-kserve-workload-svc
23:16:46	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.HTTPRoute llm/grid-inv-395bb97b-model-kserve-route
23:16:47	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.NetworkPolicy llm/grid-inv-395bb97b-model-prometheus-scraping
23:16:47	llm	multus	grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
23:16:47	llm	kubelet	grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
23:16:47	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.PodMonitor llm/grid-inv-395bb97b-model-kserve-llmisvc-engine
23:16:47	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Created	Created v1.PodMonitor llm/grid-inv-395bb97b-model-kserve-llmisvc-engine-default
23:16:48	llm	kubelet	grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	Created	Container created
23:16:48	llm	kubelet	grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	Started	Container started
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.PodMonitor llm/grid-inv-395bb97b-unmatched-kserve-llmisvc-engine
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.Deployment llm/grid-inv-395bb97b-unmatched-kserve
23:16:49	llm	replicaset-controller	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff	SuccessfulCreate	Created pod: grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.PodMonitor llm/grid-inv-395bb97b-unmatched-kserve-llmisvc-engine-default
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.DestinationRule llm/grid-inv-395bb97b-unmatched-kserve-workload-svc
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-395bb97b-unmatched-kserve-route
23:16:49	llm	deployment-controller	grid-inv-395bb97b-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-395bb97b-unmatched-kserve-7b88dd49ff from 0 to 1
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.Secret llm/grid-inv-395bb97b-unmatched-kserve-self-signed-certs
23:16:49	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.Service llm/grid-inv-395bb97b-unmatched-kserve-workload-svc
23:16:50	llm	kubelet	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Started	Container started
23:16:50	llm	kubelet	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Created	Container created
23:16:50	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	Updated	Updated v1.Secret llm/grid-inv-395bb97b-model-kserve-self-signed-certs
23:16:50	llm	multus	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
23:16:50	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-395bb97b-unmatched-prometheus-scraping
23:16:50	llm	kubelet	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
23:16:53	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	Updated	Updated v1.Secret llm/grid-inv-395bb97b-unmatched-kserve-self-signed-certs
23:16:57	llm	kubelet	grid-inv-a485b453-model-kserve-68cb9dd9c6-q4xdf	Unhealthy	Readiness probe failed: Get "https://10.132.0.58:8000/ready": dial tcp 10.132.0.58:8000: connect: connection refused
23:16:57	llm	LLMInferenceServiceController	grid-inv-395bb97b-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-395bb97b-model] is Ready
23:17:01	llm	LLMInferenceServiceController	grid-inv-395bb97b-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-395bb97b-unmatched] is Ready
23:17:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	ProbeError	Readiness probe error: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:17:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:17:52 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	ProbeError	Readiness probe error: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:17:52 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Unhealthy	Readiness probe failed: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:17:53 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Unhealthy	Readiness probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:17:53 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	ProbeError	Readiness probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:09	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Unhealthy	Readiness probe failed: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:18:09	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	ProbeError	Readiness probe error: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:19 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	ProbeError	Readiness probe error: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:19 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	ProbeError	Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:19 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	ProbeError	Readiness probe error: Get "http://10.133.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:19 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Unhealthy	Readiness probe failed: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:18:19 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Unhealthy	Readiness probe failed: Get "http://10.133.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:18:19 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	ProbeError	Readiness probe error: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
23:18:19 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-7q6zk	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:18:19 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Unhealthy	Readiness probe failed: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:18:27	llm	kubelet	grid-inv-395bb97b-model-kserve-59fb55d785-bhcqk	Killing	Stopping container main
23:18:30	llm	kubelet	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Killing	Stopping container main
23:18:45	llm	kubelet	grid-inv-395bb97b-unmatched-kserve-7b88dd49ff-zhp7q	Unhealthy	Readiness probe failed: Get "https://10.134.0.61:8000/ready": dial tcp 10.134.0.61:8000: connect: connection refused
23:19:06	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-70a7bf-555878474-6vls5	Killing	Stopping container payload-processing
23:19:06	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-70a7bf-5d8fd6cddch7tzs	Killing	Stopping container payload-pre-processing
23:19:13	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Killing	Stopping container istio-proxy
23:19:15	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
23:19:15	openshift-ingress	kubelet	e2e-shared-a-main-70a7bf-openshift-default-f9b56f4df-klf9n	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
23:19:19	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-8235df-6854bffd47bmbx2	Killing	Stopping container payload-pre-processing
23:19:19	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-8235df-565c44d86-hsh9b	Killing	Stopping container payload-processing
23:19:26	openshift-ingress	kubelet	e2e-shared-b-main-8235df-openshift-default-7b66c97d45-t2m7b	Killing	Stopping container istio-proxy
23:24:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-gpnwl
23:24:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gpnwl	Killing	Stopping container istio-proxy
23:24:10	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-jtf6f	Killing	Stopping container istio-proxy
23:24:10	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-68cb9b4b95 from 10 to 8
23:24:10	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 8; reason: All metrics below target
23:24:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-jtf6f
23:24:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-4wxx9	Killing	Stopping container istio-proxy
23:24:25	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-4wxx9
23:24:25	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-gxm54
23:24:25	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-68cb9b4b95 from 8 to 6
23:24:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-gxm54	Killing	Stopping container istio-proxy
23:24:25	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 6; reason: All metrics below target
23:24:40	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-g9ntf	Killing	Stopping container istio-proxy
23:24:40	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-55s9l	Killing	Stopping container istio-proxy
23:24:40	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-g9ntf
23:24:40	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-pf29j	Killing	Stopping container istio-proxy
23:24:40	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-pf29j
23:24:40	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-68cb9b4b95 from 6 to 3
23:24:40	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 3; reason: All metrics below target
23:24:40	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-55s9l
23:24:55	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 1; reason: All metrics below target
23:24:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-ddn95	Killing	Stopping container istio-proxy
23:24:55	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-kjjnb
23:24:55	openshift-ingress	kubelet	maas-default-gateway-openshift-default-68cb9b4b95-kjjnb	Killing	Stopping container istio-proxy
23:24:55	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-68cb9b4b95 from 3 to 1
23:24:55	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-68cb9b4b95	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-68cb9b4b95-ddn95