Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:56:16	kuadrant-system		limitador-operator-controller-manager-5778bb4449-2sxcf	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-5778bb4449-2sxcf to ip-10-0-131-64.us-west-1.compute.internal
19:44:13	openshift-ingress		router-default-85d7978dc5-vflbf	FailedScheduling	no nodes available to schedule pods
19:56:34	cert-manager		cert-manager-597b95cb84-p6mq6	Scheduled	Successfully assigned cert-manager/cert-manager-597b95cb84-p6mq6 to ip-10-0-131-101.us-west-1.compute.internal
19:57:03	openshift-jobset-operator		jobset-controller-manager-f755d7bcd-66vwk	Scheduled	Successfully assigned openshift-jobset-operator/jobset-controller-manager-f755d7bcd-66vwk to ip-10-0-131-101.us-west-1.compute.internal
19:56:31	cert-manager		cert-manager-cainjector-5c57bbd4fc-92nwt	Scheduled	Successfully assigned cert-manager/cert-manager-cainjector-5c57bbd4fc-92nwt to ip-10-0-131-101.us-west-1.compute.internal
20:02:21	llm		facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	Scheduled	Successfully assigned llm/facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t to ip-10-0-131-64.us-west-1.compute.internal
19:58:44	opendatahub		maas-controller-64ddf68ff-xlhkx	Scheduled	Successfully assigned opendatahub/maas-controller-64ddf68ff-xlhkx to ip-10-0-134-71.us-west-1.compute.internal
19:58:07	opendatahub		llmisvc-controller-manager-869d5dcc5d-b786t	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-b786t to ip-10-0-131-101.us-west-1.compute.internal
20:00:23	opendatahub		llmisvc-controller-manager-6466c554d8-7b7lk	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-7b7lk to ip-10-0-131-101.us-west-1.compute.internal
19:56:59	opendatahub		kserve-module-controller-manager-7566cf5fc8-6hch7	Scheduled	Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-6hch7 to ip-10-0-131-64.us-west-1.compute.internal
19:58:06	opendatahub		kserve-controller-manager-776cc66d47-b8vcq	Scheduled	Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-b8vcq to ip-10-0-131-101.us-west-1.compute.internal
20:01:16	opendatahub		ai-gateway-controller-5d8dddb676-xsc87	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-5d8dddb676-xsc87 to ip-10-0-134-71.us-west-1.compute.internal
20:01:34	opendatahub		ai-gateway-controller-598949b94-lwknj	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-598949b94-lwknj to ip-10-0-134-71.us-west-1.compute.internal
20:06:54	openshift-ingress		payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494 to ip-10-0-131-64.us-west-1.compute.internal
19:56:08	kuadrant-system		authorino-operator-7b7bb4bc77-pdd6r	Scheduled	Successfully assigned kuadrant-system/authorino-operator-7b7bb4bc77-pdd6r to ip-10-0-131-64.us-west-1.compute.internal
19:56:24	cert-manager		cert-manager-webhook-5dbbbf75cc-699m8	Scheduled	Successfully assigned cert-manager/cert-manager-webhook-5dbbbf75cc-699m8 to ip-10-0-131-101.us-west-1.compute.internal
20:03:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-lgvxj to ip-10-0-134-71.us-west-1.compute.internal
20:04:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-n79ks	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n79ks to ip-10-0-131-64.us-west-1.compute.internal
19:58:45	opendatahub		maas-controller-8f6976955-8fvl7	Scheduled	Successfully assigned opendatahub/maas-controller-8f6976955-8fvl7 to ip-10-0-134-71.us-west-1.compute.internal
19:58:07	opendatahub		model-serving-api-75bd89fdfc-fv5qg	Scheduled	Successfully assigned opendatahub/model-serving-api-75bd89fdfc-fv5qg to ip-10-0-131-101.us-west-1.compute.internal
19:58:07	opendatahub		odh-model-controller-769875db56-f52jn	Scheduled	Successfully assigned opendatahub/odh-model-controller-769875db56-f52jn to ip-10-0-134-71.us-west-1.compute.internal
20:04:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-gvnng	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gvnng to ip-10-0-131-64.us-west-1.compute.internal
20:02:40	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6hkkd to ip-10-0-134-71.us-west-1.compute.internal
20:04:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2x8vt to ip-10-0-131-101.us-west-1.compute.internal
20:10:42	llm		grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	Scheduled	Successfully assigned llm/grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk to ip-10-0-134-71.us-west-1.compute.internal
19:55:21	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-q6cx2 to ip-10-0-131-64.us-west-1.compute.internal
20:02:04	llm		e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	Scheduled	Successfully assigned llm/e2e-distinct-simulated-kserve-7b998b79d9-2v7cp to ip-10-0-134-71.us-west-1.compute.internal
20:03:55	llm		grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	Scheduled	Successfully assigned llm/grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx to ip-10-0-131-64.us-west-1.compute.internal
19:56:53	kuadrant-system		authorino-56c6c9ddd5-snmh8	Scheduled	Successfully assigned kuadrant-system/authorino-56c6c9ddd5-snmh8 to ip-10-0-134-71.us-west-1.compute.internal
20:02:07	llm		e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	Scheduled	Successfully assigned llm/e2e-embedding-simulated-kserve-67bf594cd6-ttrwc to ip-10-0-131-64.us-west-1.compute.internal
20:03:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-tqlvx to ip-10-0-131-101.us-west-1.compute.internal
20:02:55	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xj5qx to ip-10-0-131-64.us-west-1.compute.internal
19:56:46	openshift-ingress		kube-auth-proxy-655c576ffb-lvlq5	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-655c576ffb-lvlq5 to ip-10-0-131-101.us-west-1.compute.internal
20:08:56	opendatahub		test-curl-9041-0ce8b3	Scheduled	Successfully assigned opendatahub/test-curl-9041-0ce8b3 to ip-10-0-134-71.us-west-1.compute.internal
20:02:55	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xtsh2 to ip-10-0-131-101.us-west-1.compute.internal
19:56:47	openshift-ingress		data-science-gateway-data-science-gateway-class-76f56c5945fk5rx	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945fk5rx to ip-10-0-131-64.us-west-1.compute.internal
20:02:25	llm		premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	Scheduled	Successfully assigned llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb to ip-10-0-134-71.us-west-1.compute.internal
19:56:46	openshift-ingress		kube-auth-proxy-655c576ffb-dbtgj	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-655c576ffb-dbtgj to ip-10-0-134-71.us-west-1.compute.internal
19:56:50	kuadrant-system		authorino-57f7f8659f-5dwjk	Scheduled	Successfully assigned kuadrant-system/authorino-57f7f8659f-5dwjk to ip-10-0-131-101.us-west-1.compute.internal
20:10:46	llm		grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	Scheduled	Successfully assigned llm/grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z to ip-10-0-134-71.us-west-1.compute.internal
19:56:11	opendatahub-operator-system		opendatahub-operator-controller-manager-756fd67b76-ln277	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-756fd67b76-ln277 to ip-10-0-131-64.us-west-1.compute.internal
20:02:55	kuadrant-system		authorino-d8886d8d8-cfg9x	Scheduled	Successfully assigned kuadrant-system/authorino-d8886d8d8-cfg9x to ip-10-0-134-71.us-west-1.compute.internal
19:58:01	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-86dq7	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-86dq7 to ip-10-0-134-71.us-west-1.compute.internal
19:44:15	openshift-ingress		router-default-85d7978dc5-vflbf	FailedScheduling	no nodes available to schedule pods
20:01:57	llm		e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	Scheduled	Successfully assigned llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6 to ip-10-0-134-71.us-west-1.compute.internal
20:05:59	llm		grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	Scheduled	Successfully assigned llm/grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8 to ip-10-0-134-71.us-west-1.compute.internal
19:55:22	openshift-ingress		openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm to ip-10-0-131-64.us-west-1.compute.internal
19:59:42	kuadrant-system		authorino-67fb44bc55-2x7gn	Scheduled	Successfully assigned kuadrant-system/authorino-67fb44bc55-2x7gn to ip-10-0-134-71.us-west-1.compute.internal
20:14:05	ai-tenants		e2e-shared-b-main-12288c	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-b-main-12288c: MaasTenantConfig is not available to restore revocation operands
20:07:24	openshift-ingress		payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w to ip-10-0-134-71.us-west-1.compute.internal
20:04:10	openshift-ingress		maas-default-gateway-openshift-default-59f957fdd7-kndtt	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kndtt to ip-10-0-134-71.us-west-1.compute.internal
20:01:27	openshift-ingress		payload-processing-7bb9489676-qcnxp	Scheduled	Successfully assigned openshift-ingress/payload-processing-7bb9489676-qcnxp to ip-10-0-134-71.us-west-1.compute.internal
19:56:11	opendatahub-operator-system		opendatahub-operator-controller-manager-756fd67b76-2h68q	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-756fd67b76-2h68q to ip-10-0-131-101.us-west-1.compute.internal
19:56:12	kuadrant-system		dns-operator-controller-manager-8557cbdb84-8xczb	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xczb to ip-10-0-131-101.us-west-1.compute.internal
19:55:13	openshift-ingress		istiod-openshift-gateway-7876bfcb74-gnvm4	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-gnvm4 to ip-10-0-134-71.us-west-1.compute.internal
19:58:56	opendatahub		usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	Scheduled	Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hcbgc to ip-10-0-134-71.us-west-1.compute.internal
19:59:41	kuadrant-system		authorino-6f6944bcdd-l8dlh	Scheduled	Successfully assigned kuadrant-system/authorino-6f6944bcdd-l8dlh to ip-10-0-134-71.us-west-1.compute.internal
20:07:16	openshift-ingress		e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r to ip-10-0-131-64.us-west-1.compute.internal
19:56:53	kuadrant-system		limitador-limitador-7c9b87d574-tr4wk	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-7c9b87d574-tr4wk to ip-10-0-134-71.us-west-1.compute.internal
20:06:46	openshift-ingress		e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8 to ip-10-0-131-64.us-west-1.compute.internal
20:03:32	llm		grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	Scheduled	Successfully assigned llm/grid-inv-b695bb2e-model-kserve-6dc694749-t5m85 to ip-10-0-134-71.us-west-1.compute.internal
20:02:10	llm		e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	Scheduled	Successfully assigned llm/e2e-trlp-test-simulated-kserve-69b588897d-rfs4d to ip-10-0-134-71.us-west-1.compute.internal
20:06:38	opendatahub		test-curl-9041-a678b6	Scheduled	Successfully assigned opendatahub/test-curl-9041-a678b6 to ip-10-0-131-64.us-west-1.compute.internal
19:56:11	opendatahub-operator-system		opendatahub-operator-controller-manager-756fd67b76-xg7mx	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-756fd67b76-xg7mx to ip-10-0-134-71.us-west-1.compute.internal
19:56:49	kuadrant-system		limitador-limitador-74dfcfb894-brl7b	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-74dfcfb894-brl7b to ip-10-0-131-64.us-west-1.compute.internal
20:00:17	kuadrant-system		authorino-7d8bb9f59f-xv9s5	Scheduled	Successfully assigned kuadrant-system/authorino-7d8bb9f59f-xv9s5 to ip-10-0-134-71.us-west-1.compute.internal
20:13:44	ai-tenants		e2e-shared-a-main-e56f6e	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-a-main-e56f6e: MaasTenantConfig is not available to restore revocation operands
19:56:25	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-85txq	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-85txq to ip-10-0-131-64.us-west-1.compute.internal
20:01:27	openshift-ingress		payload-pre-processing-7b98bb678c-wpsp2	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-wpsp2 to ip-10-0-134-71.us-west-1.compute.internal
20:02:14	llm		e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	Scheduled	Successfully assigned llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5 to ip-10-0-131-64.us-west-1.compute.internal
20:06:32	opendatahub		test-curl-9041-2c82f4	Scheduled	Successfully assigned opendatahub/test-curl-9041-2c82f4 to ip-10-0-131-64.us-west-1.compute.internal
19:50:40	openshift-ingress		router-default-85d7978dc5-vflbf	Scheduled	Successfully assigned openshift-ingress/router-default-85d7978dc5-vflbf to ip-10-0-131-64.us-west-1.compute.internal
19:56:27	openshift-jobset-operator		jobset-operator-674cd59dc5-dngzl	Scheduled	Successfully assigned openshift-jobset-operator/jobset-operator-674cd59dc5-dngzl to ip-10-0-131-101.us-west-1.compute.internal
20:02:18	llm		echo-simulator-kserve-64b6f8f96c-6rx96	Scheduled	Successfully assigned llm/echo-simulator-kserve-64b6f8f96c-6rx96 to ip-10-0-134-71.us-west-1.compute.internal
19:59:41	kuadrant-system		authorino-d6cffbf65-vwchz	Scheduled	Successfully assigned kuadrant-system/authorino-d6cffbf65-vwchz to ip-10-0-134-71.us-west-1.compute.internal
20:06:03	llm		grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	Scheduled	Successfully assigned llm/grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg to ip-10-0-134-71.us-west-1.compute.internal
19:56:36	kuadrant-system		kuadrant-console-plugin-66c4fb575d-cswfr	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-66c4fb575d-cswfr to ip-10-0-131-101.us-west-1.compute.internal
20:06:54	openshift-ingress		payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l to ip-10-0-131-64.us-west-1.compute.internal
20:06:42	opendatahub		test-curl-9041-19b575	Scheduled	Successfully assigned opendatahub/test-curl-9041-19b575 to ip-10-0-131-64.us-west-1.compute.internal
20:07:24	openshift-ingress		payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g to ip-10-0-131-64.us-west-1.compute.internal
19:44:13	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-85d7978dc5 from 0 to 1
19:44:13	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
19:44:13	openshift-ingress	replicaset-controller	router-default-85d7978dc5	SuccessfulCreate	Created pod: router-default-85d7978dc5-vflbf
19:44:13	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
19:44:18	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
19:50:25 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
19:50:48 (x5)	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
19:50:48 (x5)	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
19:50:54 (x2)	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
19:50:55 (x3)	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
19:50:57	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	Started	Container started
19:50:57	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
19:50:57	openshift-ingress	kubelet	router-default-85d7978dc5-vflbf	Created	Container created
19:50:57	openshift-ingress	multus	router-default-85d7978dc5-vflbf	AddedInterface	Add eth0 [10.132.0.19/23] from ovn-kubernetes
19:55:11	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
19:55:13	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-gnvm4	AddedInterface	Add eth0 [10.133.0.16/23] from ovn-kubernetes
19:55:13	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
19:55:13	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-gnvm4
19:55:13	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
19:55:18	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 5.007s (5.007s including waiting). Image size: 185476477 bytes.
19:55:18	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Created	Container created
19:55:18	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Started	Container started
19:55:20 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:55:20 (x2)	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
19:55:21	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-q6cx2
19:55:21	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-59f957fdd7 from 0 to 1
19:55:22	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-7cd79c685f from 0 to 1
19:55:22	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
19:55:22	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-7cd79c685f	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm
19:55:22	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
19:55:22	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
19:55:22	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
19:55:22	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
19:55:26	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
19:55:34	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 11.874s (11.874s including waiting). Image size: 183555333 bytes.
19:55:34	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 11.327s (11.327s including waiting). Image size: 183555333 bytes.
19:55:34	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	Created	Container created
19:55:34	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-7cd79c685f-k9wnm	Started	Container started
19:55:52 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:52 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:03	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
19:56:03	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
19:56:07	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
19:56:07	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
19:56:07	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:07	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:08	kuadrant-system	replicaset-controller	authorino-operator-7b7bb4bc77	SuccessfulCreate	Created pod: authorino-operator-7b7bb4bc77-pdd6r
19:56:08	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
19:56:08	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
19:56:08	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-pdd6r	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589"
19:56:08	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-7b7bb4bc77 from 0 to 1
19:56:08	kuadrant-system	multus	authorino-operator-7b7bb4bc77-pdd6r	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
19:56:08	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
19:56:09	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsUnknown	requirements not yet checked
19:56:09	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsNotMet	one or more requirements couldn't be found
19:56:11	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-756fd67b76	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-756fd67b76-2h68q
19:56:11	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-756fd67b76	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-756fd67b76-ln277
19:56:11	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-756fd67b76	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-756fd67b76-xg7mx
19:56:11	opendatahub-operator-system	deployment-controller	opendatahub-operator-controller-manager	ScalingReplicaSet	Scaled up replica set opendatahub-operator-controller-manager-756fd67b76 from 0 to 3
19:56:11	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	AllRequirementsMet	all requirements found, attempting install
19:56:12	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
19:56:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:12	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-pdd6r	Created	Container created
19:56:12	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-pdd6r	Started	Container started
19:56:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-756fd67b76-ln277	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
19:56:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:12	kuadrant-system	multus	dns-operator-controller-manager-8557cbdb84-8xczb	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
19:56:12	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
19:56:12	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
19:56:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-756fd67b76-2h68q	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
19:56:12	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-8557cbdb84 from 0 to 1
19:56:12	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
19:56:12	kuadrant-system	replicaset-controller	dns-operator-controller-manager-8557cbdb84	SuccessfulCreate	Created pod: dns-operator-controller-manager-8557cbdb84-8xczb
19:56:12	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-pdd6r	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589" in 3.707s (3.707s including waiting). Image size: 167738465 bytes.
19:56:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-756fd67b76-xg7mx	AddedInterface	Add eth0 [10.133.0.17/23] from ovn-kubernetes
19:56:12	kuadrant-system	authorino-operator-7b7bb4bc77-pdd6r_ccd6aece-adc7-41be-9c6d-5992bb30afc4	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-7b7bb4bc77-pdd6r_ccd6aece-adc7-41be-9c6d-5992bb30afc4 became leader
19:56:13	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-8xczb	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a"
19:56:13	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Started	Container started
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Created	Container created
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Started	Container started
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 3.414s (3.414s including waiting). Image size: 273709574 bytes.
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 3.567s (3.567s including waiting). Image size: 273709574 bytes.
19:56:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Created	Container created
19:56:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 4.332s (4.332s including waiting). Image size: 273709574 bytes.
19:56:16	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
19:56:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:16	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
19:56:16	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
19:56:16	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-5778bb4449	SuccessfulCreate	Created pod: limitador-operator-controller-manager-5778bb4449-2sxcf
19:56:16	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-5778bb4449 from 0 to 1
19:56:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Created	Container created
19:56:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Started	Container started
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 791ms (791ms including waiting). Image size: 273709574 bytes.
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Created	Container created
19:56:17	kuadrant-system	multus	limitador-operator-controller-manager-5778bb4449-2sxcf	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
19:56:17	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
19:56:17	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-2sxcf	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e"
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Started	Container started
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 827ms (827ms including waiting). Image size: 273709574 bytes.
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Started	Container started
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-2h68q	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:17	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-xg7mx	Created	Container created
19:56:18	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:56:18	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
19:56:18	opendatahub-operator-system	opendatahub-operator-controller-manager-756fd67b76-xg7mx_a3ca9094-9407-415a-9cd2-fbaf002aec4a	07ed84f7.opendatahub.io	LeaderElection	opendatahub-operator-controller-manager-756fd67b76-xg7mx_a3ca9094-9407-415a-9cd2-fbaf002aec4a became leader
19:56:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Created	Container created
19:56:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Started	Container started
19:56:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-756fd67b76-ln277	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 849ms (849ms including waiting). Image size: 273709574 bytes.
19:56:19	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-2sxcf	Started	Container started
19:56:19	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-2sxcf	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e" in 2.521s (2.521s including waiting). Image size: 177967631 bytes.
19:56:19	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-2sxcf	Created	Container created
19:56:20	opendatahub-operator-system	operator-lifecycle-manager	install-lkcq9	AppliedWithWarnings	1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1
19:56:20	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	Webhook install failed: conversionWebhook not ready
19:56:20	kuadrant-system	limitador-operator-controller-manager-5778bb4449-2sxcf_2aa73b97-53ec-4395-be4c-b45d35b01bbb	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-5778bb4449-2sxcf_2aa73b97-53ec-4395-be4c-b45d35b01bbb became leader
19:56:21	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-8xczb	Started	Container started
19:56:21 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	waiting for install components to report healthy
19:56:21	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-8xczb	Created	Container created
19:56:21	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-8xczb	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a" in 8.817s (8.817s including waiting). Image size: 190804149 bytes.
19:56:22	kuadrant-system	dns-operator-controller-manager-8557cbdb84-8xczb_68fff359-1b27-455c-8e50-4e14657ad8e3	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-8557cbdb84-8xczb_68fff359-1b27-455c-8e50-4e14657ad8e3 became leader
19:56:22 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability.
19:56:24	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
19:56:24	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
19:56:24	cert-manager	deployment-controller	cert-manager	ScalingReplicaSet	Scaled up replica set cert-manager-597b95cb84 from 0 to 1
19:56:24	cert-manager	deployment-controller	cert-manager-webhook	ScalingReplicaSet	Scaled up replica set cert-manager-webhook-5dbbbf75cc from 0 to 1
19:56:24	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	SuccessfulCreate	Created pod: cert-manager-webhook-5dbbbf75cc-699m8
19:56:24 (x6)	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	FailedCreate	Error creating: pods "cert-manager-webhook-5dbbbf75cc-" is forbidden: error looking up service account cert-manager/cert-manager-webhook: serviceaccount "cert-manager-webhook" not found
19:56:25	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-678d87766f from 0 to 1
19:56:25	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
19:56:25	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-85txq
19:56:25	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-699m8	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
19:56:25	cert-manager	multus	cert-manager-webhook-5dbbbf75cc-699m8	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:56:26	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	AllRequirementsMet	all requirements found, attempting install
19:56:26	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsUnknown	requirements not yet checked
19:56:26	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
19:56:26	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-85txq	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"
19:56:26	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-85txq	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
19:56:27	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	waiting for install components to report healthy
19:56:27	openshift-jobset-operator	deployment-controller	jobset-operator	ScalingReplicaSet	Scaled up replica set jobset-operator-674cd59dc5 from 0 to 1
19:56:27	openshift-jobset-operator	replicaset-controller	jobset-operator-674cd59dc5	SuccessfulCreate	Created pod: jobset-operator-674cd59dc5-dngzl
19:56:27	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-dngzl	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e"
19:56:27	openshift-jobset-operator	multus	jobset-operator-674cd59dc5-dngzl	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
19:56:27	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallWaiting	installing: waiting for deployment jobset-operator to become ready: deployment "jobset-operator" not available: Deployment does not have minimum availability.
19:56:29	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-699m8	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 4.393s (4.393s including waiting). Image size: 309650901 bytes.
19:56:29 (x11)	cert-manager	replicaset-controller	cert-manager-597b95cb84	FailedCreate	Error creating: pods "cert-manager-597b95cb84-" is forbidden: error looking up service account cert-manager/cert-manager: serviceaccount "cert-manager" not found
19:56:29	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-699m8	Created	Container created
19:56:29	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-699m8	Started	Container started
19:56:31	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-dngzl	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e" in 3.693s (3.693s including waiting). Image size: 224874686 bytes.
19:56:31	cert-manager	deployment-controller	cert-manager-cainjector	ScalingReplicaSet	Scaled up replica set cert-manager-cainjector-5c57bbd4fc from 0 to 1
19:56:31	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	SuccessfulCreate	Created pod: cert-manager-cainjector-5c57bbd4fc-92nwt
19:56:31	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	install strategy completed with no errors
19:56:31	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-dngzl	Created	Container created
19:56:31	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-dngzl	Started	Container started
19:56:32	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-92nwt	Pulled	Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod
19:56:32	cert-manager	multus	cert-manager-cainjector-5c57bbd4fc-92nwt	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
19:56:32	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	install strategy completed with no errors
19:56:32	openshift-jobset-operator	openshift-jobset-operator	openshift-jobset-operator-lock	LeaderElection	jobset-operator-674cd59dc5-dngzl_19c73aa2-9200-4eb4-9245-5beaac635466 became leader
19:56:32	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-92nwt	Started	Container started
19:56:32	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-85txq_ddacba38-e466-4d4f-9c31-8504df421cd0	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-85txq_ddacba38-e466-4d4f-9c31-8504df421cd0 became leader
19:56:32	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-85txq	Started	Container started
19:56:32	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-85txq	Created	Container created
19:56:32	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-85txq	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 5.648s (5.648s including waiting). Image size: 424244516 bytes.
19:56:32	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-92nwt	Created	Container created
19:56:32	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
19:56:33	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
19:56:34	cert-manager	replicaset-controller	cert-manager-597b95cb84	SuccessfulCreate	Created pod: cert-manager-597b95cb84-p6mq6
19:56:35	cert-manager	kubelet	cert-manager-597b95cb84-p6mq6	Created	Container created
19:56:35	cert-manager	multus	cert-manager-597b95cb84-p6mq6	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
19:56:35	cert-manager	kubelet	cert-manager-597b95cb84-p6mq6	Pulled	Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod
19:56:35	cert-manager	kubelet	cert-manager-597b95cb84-p6mq6	Started	Container started
19:56:36	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-66c4fb575d from 0 to 1
19:56:36	kuadrant-system	replicaset-controller	kuadrant-console-plugin-66c4fb575d	SuccessfulCreate	Created pod: kuadrant-console-plugin-66c4fb575d-cswfr
19:56:37	kuadrant-system	multus	kuadrant-console-plugin-66c4fb575d-cswfr	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
19:56:37	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-cswfr	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044"
19:56:43	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
19:56:44	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceAccountCreated	Created ServiceAccount/jobset-controller-manager -n openshift-jobset-operator because it was missing
19:56:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-cswfr	Created	Container created
19:56:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-cswfr	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044" in 7.241s (7.241s including waiting). Image size: 342538866 bytes.
19:56:44	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-cswfr	Started	Container started
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-serving-cert -n openshift-jobset-operator because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-metrics-reader because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceCreated	Created Service/jobset-webhook-service -n openshift-jobset-operator because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-manager-role because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	IssuerCreated	Created Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-manager-rolebinding because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-proxy-role because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-leader-election-role -n openshift-jobset-operator because it was missing
19:56:45	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-metrics-cert -n openshift-jobset-operator because it was missing
19:56:46	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-655c576ffb from 0 to 2
19:56:46	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-manager-secrets-role -n openshift-jobset-operator because it was missing
19:56:46	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding -n openshift-jobset-operator because it was missing
19:56:46	openshift-ingress	replicaset-controller	kube-auth-proxy-655c576ffb	SuccessfulCreate	Created pod: kube-auth-proxy-655c576ffb-lvlq5
19:56:46	openshift-ingress	replicaset-controller	kube-auth-proxy-655c576ffb	SuccessfulCreate	Created pod: kube-auth-proxy-655c576ffb-dbtgj
19:56:47	openshift-ingress	multus	kube-auth-proxy-655c576ffb-dbtgj	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
19:56:47	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-dbtgj	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
19:56:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-manager-secrets-rolebinding -n openshift-jobset-operator because it was missing
19:56:47	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-76f56c5945	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-76f56c5945fk5rx
19:56:47	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceCreated	Created Service/jobset-controller-manager-metrics-service -n openshift-jobset-operator because it was missing
19:56:47	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-76f56c5945 from 0 to 1
19:56:47	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-lvlq5	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
19:56:47	openshift-ingress	multus	kube-auth-proxy-655c576ffb-lvlq5	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:56:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-76f56c5945fk5rx	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
19:56:48	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-76f56c5945fk5rx	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
19:56:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-76f56c5945fk5rx	Created	Container created
19:56:48	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-76f56c5945fk5rx	Started	Container started
19:56:49	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-74dfcfb894 from 0 to 1
19:56:49	kuadrant-system	multus	limitador-limitador-74dfcfb894-brl7b	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:56:49	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-brl7b	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
19:56:49	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulCreate	Created pod: limitador-limitador-74dfcfb894-brl7b
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-dbtgj	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.87s (2.87s including waiting). Image size: 188904677 bytes.
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-dbtgj	Created	Container created
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-dbtgj	Started	Container started
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-lvlq5	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.995s (2.995s including waiting). Image size: 188904677 bytes.
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-lvlq5	Created	Container created
19:56:50	openshift-ingress	kubelet	kube-auth-proxy-655c576ffb-lvlq5	Started	Container started
19:56:50	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulCreate	Created pod: authorino-57f7f8659f-5dwjk
19:56:50	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-57f7f8659f from 0 to 1
19:56:51	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-brl7b	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 1.94s (1.94s including waiting). Image size: 133379307 bytes.
19:56:51	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-brl7b	Started	Container started
19:56:51	kuadrant-system	kubelet	authorino-57f7f8659f-5dwjk	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:56:51	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-brl7b	Created	Container created
19:56:51	kuadrant-system	multus	authorino-57f7f8659f-5dwjk	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
19:56:53	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-7c9b87d574 from 0 to 1
19:56:53	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulCreate	Created pod: authorino-56c6c9ddd5-snmh8
19:56:53	kuadrant-system	replicaset-controller	limitador-limitador-7c9b87d574	SuccessfulCreate	Created pod: limitador-limitador-7c9b87d574-tr4wk
19:56:53	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-56c6c9ddd5 from 0 to 1
19:56:54	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-tr4wk	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
19:56:54	kuadrant-system	multus	limitador-limitador-7c9b87d574-tr4wk	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
19:56:54	kuadrant-system	kubelet	authorino-56c6c9ddd5-snmh8	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:56:54	kuadrant-system	multus	authorino-56c6c9ddd5-snmh8	AddedInterface	Add eth0 [10.133.0.19/23] from ovn-kubernetes
19:56:55	kuadrant-system	kubelet	authorino-57f7f8659f-5dwjk	Created	Container created
19:56:55	kuadrant-system	kubelet	authorino-57f7f8659f-5dwjk	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 4.302s (4.302s including waiting). Image size: 204096810 bytes.
19:56:55	kuadrant-system	kubelet	authorino-57f7f8659f-5dwjk	Started	Container started
19:56:57	kuadrant-system	kubelet	authorino-56c6c9ddd5-snmh8	Created	Container created
19:56:57	kuadrant-system	kubelet	authorino-56c6c9ddd5-snmh8	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 3.49s (3.49s including waiting). Image size: 204096810 bytes.
19:56:57	kuadrant-system	kubelet	authorino-56c6c9ddd5-snmh8	Started	Container started
19:56:57	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-tr4wk	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 3.288s (3.288s including waiting). Image size: 133379307 bytes.
19:56:57	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-tr4wk	Created	Container created
19:56:57	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-tr4wk	Started	Container started
19:56:58	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulDelete	Deleted pod: authorino-57f7f8659f-5dwjk
19:56:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-57f7f8659f from 1 to 0
19:56:58	kuadrant-system	kubelet	authorino-57f7f8659f-5dwjk	Killing	Stopping container authorino
19:56:59	opendatahub	deployment-controller	kserve-module-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1
19:56:59	opendatahub	replicaset-controller	kserve-module-controller-manager-7566cf5fc8	SuccessfulCreate	Created pod: kserve-module-controller-manager-7566cf5fc8-6hch7
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-metrics-cert	Requested	Created new CertificateRequest resource "jobset-metrics-cert-1"
19:57:00	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-metrics-cert	Generated	Stored new private key in temporary Secret resource "jobset-metrics-cert-smfcw"
19:57:00	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-metrics-cert	Issuing	Issuing certificate as Secret does not exist
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	opendatahub	multus	kserve-module-controller-manager-7566cf5fc8-6hch7	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:57:00	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-6hch7	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189"
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:57:00	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-serving-cert	Requested	Created new CertificateRequest resource "jobset-serving-cert-1"
19:57:00	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-serving-cert	Issuing	Issuing certificate as Secret does not exist
19:57:00	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-serving-cert	Generated	Stored new private key in temporary Secret resource "jobset-serving-cert-ghj7z"
19:57:01	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	CertificateIssued	Certificate fetched from issuer successfully
19:57:01	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-metrics-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
19:57:01	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-serving-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
19:57:01	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-metrics-cert	Issuing	The certificate has been successfully issued
19:57:01	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	CertificateIssued	Certificate fetched from issuer successfully
19:57:01	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationCreated	Created ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it was missing
19:57:01	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-serving-cert	Issuing	The certificate has been successfully issued
19:57:02	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:02	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:02	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CustomResourceDefinitionCreated	Created CustomResourceDefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io because it was missing
19:57:02	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationCreated	Created MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it was missing
19:57:02	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
19:57:02	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
19:57:03	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-6hch7	Created	Container created
19:57:03	openshift-jobset-operator	deployment-controller	jobset-controller-manager	ScalingReplicaSet	Scaled up replica set jobset-controller-manager-f755d7bcd from 0 to 1
19:57:03	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-6hch7	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.875s (2.875s including waiting). Image size: 195729000 bytes.
19:57:03	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-6hch7	Started	Container started
19:57:03	openshift-jobset-operator	replicaset-controller	jobset-controller-manager-f755d7bcd	SuccessfulCreate	Created pod: jobset-controller-manager-f755d7bcd-66vwk
19:57:03	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	DeploymentCreated	Created Deployment.apps/jobset-controller-manager -n openshift-jobset-operator because it was missing
19:57:03	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceMonitorCreated	Created ServiceMonitor.monitoring.coreos.com/jobset-controller-manager-metrics-monitor -n openshift-jobset-operator because it was missing
19:57:03	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ConfigMapCreated	Created ConfigMap/jobset-manager-config -n openshift-jobset-operator because it was missing
19:57:04	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:04	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:04	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationUpdated	Updated MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it changed
19:57:04	openshift-jobset-operator	kubelet	jobset-controller-manager-f755d7bcd-66vwk	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987"
19:57:04	openshift-jobset-operator	multus	jobset-controller-manager-f755d7bcd-66vwk	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:57:04	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationUpdated	Updated ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it changed
19:57:08	openshift-jobset-operator	kubelet	jobset-controller-manager-f755d7bcd-66vwk	Started	Container started
19:57:08	openshift-jobset-operator	kubelet	jobset-controller-manager-f755d7bcd-66vwk	Created	Container created
19:57:08	openshift-jobset-operator	kubelet	jobset-controller-manager-f755d7bcd-66vwk	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" in 3.853s (3.853s including waiting). Image size: 182184678 bytes.
19:57:08	openshift-jobset-operator	jobset-controller-manager-f755d7bcd-66vwk_b15447a3-f617-4ea1-ab41-3d473fa25656	6d4f6a47.jobset.x-k8s.io	LeaderElection	jobset-controller-manager-f755d7bcd-66vwk_b15447a3-f617-4ea1-ab41-3d473fa25656 became leader
19:57:09	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-74dfcfb894 from 1 to 0
19:57:09	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulDelete	Deleted pod: limitador-limitador-74dfcfb894-brl7b
19:57:09	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-brl7b	Killing	Stopping container limitador
19:57:17	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:17	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:19	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:19	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:20	opendatahub	kserve-module-controller-manager-7566cf5fc8-6hch7_d1b673ce-c091-40a3-bcb7-efd7ef839aa3	kserve-module-controller-manager	LeaderElection	kserve-module-controller-manager-7566cf5fc8-6hch7_d1b673ce-c091-40a3-bcb7-efd7ef839aa3 became leader
19:58:01	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-86dq7
19:58:01	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
19:58:02	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-86dq7	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
19:58:02	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-86dq7	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
19:58:02	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-86dq7	Started	Container started
19:58:02	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-86dq7	Created	Container created
19:58:03	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-86dq7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:58:03	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-86dq7	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
19:58:06	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-gnvm4	Killing	Stopping container discovery
19:58:06	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-gnvm4
19:58:06	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
19:58:06	opendatahub	replicaset-controller	kserve-controller-manager-776cc66d47	SuccessfulCreate	Created pod: kserve-controller-manager-776cc66d47-b8vcq
19:58:06	opendatahub	deployment-controller	kserve-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1
19:58:07	opendatahub	replicaset-controller	odh-model-controller-769875db56	SuccessfulCreate	Created pod: odh-model-controller-769875db56-f52jn
19:58:07	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulCreate	Created pod: llmisvc-controller-manager-869d5dcc5d-b786t
19:58:07	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1
19:58:07	opendatahub	deployment-controller	odh-model-controller	ScalingReplicaSet	Scaled up replica set odh-model-controller-769875db56 from 0 to 1
19:58:07	opendatahub	multus	llmisvc-controller-manager-869d5dcc5d-b786t	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
19:58:07	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
19:58:07	opendatahub	kubelet	kserve-controller-manager-776cc66d47-b8vcq	Pulling	Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d"
19:58:07	opendatahub	kubelet	model-serving-api-75bd89fdfc-fv5qg	Pulling	Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8"
19:58:07	opendatahub	multus	model-serving-api-75bd89fdfc-fv5qg	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:58:07	opendatahub	multus	kserve-controller-manager-776cc66d47-b8vcq	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
19:58:07	opendatahub	deployment-controller	model-serving-api	ScalingReplicaSet	Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1
19:58:07	opendatahub	replicaset-controller	model-serving-api-75bd89fdfc	SuccessfulCreate	Created pod: model-serving-api-75bd89fdfc-fv5qg
19:58:07	opendatahub	multus	odh-model-controller-769875db56-f52jn	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
19:58:08	opendatahub	kubelet	odh-model-controller-769875db56-f52jn	Pulling	Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775"
19:58:10	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Created	Container created
19:58:10	opendatahub	kubelet	model-serving-api-75bd89fdfc-fv5qg	Started	Container started
19:58:10	opendatahub	kubelet	model-serving-api-75bd89fdfc-fv5qg	Created	Container created
19:58:10	opendatahub	kubelet	model-serving-api-75bd89fdfc-fv5qg	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.449s (2.449s including waiting). Image size: 184925761 bytes.
19:58:10	opendatahub	kubelet	kserve-controller-manager-776cc66d47-b8vcq	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.759s (2.759s including waiting). Image size: 247676581 bytes.
19:58:10	opendatahub	kubelet	kserve-controller-manager-776cc66d47-b8vcq	Created	Container created
19:58:10	opendatahub	kubelet	kserve-controller-manager-776cc66d47-b8vcq	Started	Container started
19:58:10	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 2.613s (2.613s including waiting). Image size: 235100981 bytes.
19:58:10	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Started	Container started
19:58:11	opendatahub	kserve-controller-manager-776cc66d47-b8vcq_e2e1d877-01d7-447f-903d-9f66bc0f2a49	kserve-controller-manager-leader-lock	LeaderElection	kserve-controller-manager-776cc66d47-b8vcq_e2e1d877-01d7-447f-903d-9f66bc0f2a49 became leader
19:58:11	opendatahub	kubelet	odh-model-controller-769875db56-f52jn	Started	Container started
19:58:11	opendatahub	kubelet	odh-model-controller-769875db56-f52jn	Created	Container created
19:58:11	opendatahub	kubelet	odh-model-controller-769875db56-f52jn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.104s (3.104s including waiting). Image size: 236755567 bytes.
19:58:12	opendatahub	odh-model-controller-769875db56-f52jn_2749e274-42c9-4c5c-8ad9-9716c8ce2dc4	odh-model-controller.opendatahub.io	LeaderElection	odh-model-controller-769875db56-f52jn_2749e274-42c9-4c5c-8ad9-9716c8ce2dc4 became leader
19:58:12	opendatahub	llmisvc-controller-manager-869d5dcc5d-b786t_33f403bb-e9e5-4513-aa9c-fe5079c346b8	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-869d5dcc5d-b786t_33f403bb-e9e5-4513-aa9c-fe5079c346b8 became leader
19:58:29 (x5)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:29 (x5)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:44	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-64ddf68ff from 0 to 1
19:58:44	opendatahub	replicaset-controller	maas-controller-64ddf68ff	SuccessfulCreate	Created pod: maas-controller-64ddf68ff-xlhkx
19:58:45	opendatahub	replicaset-controller	maas-controller-8f6976955	SuccessfulCreate	Created pod: maas-controller-8f6976955-8fvl7
19:58:45	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-8f6976955 from 0 to 1
19:58:45	opendatahub	multus	maas-controller-8f6976955-8fvl7	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
19:58:45	opendatahub	multus	maas-controller-64ddf68ff-xlhkx	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
19:58:45	opendatahub	kubelet	maas-controller-64ddf68ff-xlhkx	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:2ba1ee5951c381efd9e0aa33e4616726dc400f92e708df5d3f6061899f078c6a"
19:58:46	opendatahub	kubelet	maas-controller-8f6976955-8fvl7	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:2ba1ee5951c381efd9e0aa33e4616726dc400f92e708df5d3f6061899f078c6a"
19:58:52	opendatahub	kubelet	maas-controller-8f6976955-8fvl7	Started	Container started
19:58:52	opendatahub	kubelet	maas-controller-8f6976955-8fvl7	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:2ba1ee5951c381efd9e0aa33e4616726dc400f92e708df5d3f6061899f078c6a" in 6.329s (6.329s including waiting). Image size: 200061437 bytes.
19:58:52	opendatahub	kubelet	maas-controller-64ddf68ff-xlhkx	Created	Container created
19:58:52	opendatahub	kubelet	maas-controller-64ddf68ff-xlhkx	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:2ba1ee5951c381efd9e0aa33e4616726dc400f92e708df5d3f6061899f078c6a" in 6.796s (6.796s including waiting). Image size: 200061437 bytes.
19:58:52	opendatahub	kubelet	maas-controller-64ddf68ff-xlhkx	Started	Container started
19:58:52	opendatahub	kubelet	maas-controller-8f6976955-8fvl7	Created	Container created
19:58:54	opendatahub	maas-controller-64ddf68ff-xlhkx_9dd42019-6e83-4aa2-8cb4-49141c72f13d	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-64ddf68ff-xlhkx_9dd42019-6e83-4aa2-8cb4-49141c72f13d became leader
19:58:56	opendatahub	deployment-controller	usage-logs-tenancy-proxy	ScalingReplicaSet	Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1
19:58:56	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	FailedMount	MountVolume.SetUp failed for volume "proxy-source" : configmap "usage-tenancy-proxy-source" not found
19:58:56	opendatahub	replicaset-controller	usage-logs-tenancy-proxy-9c5d49bcd	SuccessfulCreate	Created pod: usage-logs-tenancy-proxy-9c5d49bcd-hcbgc
19:58:56	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	FailedMount	MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found
19:58:57	opendatahub	multus	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
19:58:57	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	Pulling	Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172"
19:59:04	opendatahub	kubelet	maas-controller-64ddf68ff-xlhkx	Killing	Stopping container manager
19:59:04	opendatahub	replicaset-controller	maas-controller-64ddf68ff	SuccessfulDelete	Deleted pod: maas-controller-64ddf68ff-xlhkx
19:59:04	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled down replica set maas-controller-64ddf68ff from 1 to 0
19:59:06	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
19:59:06 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Started	Container started
19:59:06 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Created	Container created
19:59:06	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	ProbeError	Readiness probe error: Get "http://10.132.0.34:15021/healthz/ready": read tcp 10.132.0.2:52202->10.132.0.34:15021: read: connection reset by peer body:
19:59:06	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Unhealthy	Readiness probe failed: Get "http://10.132.0.34:15021/healthz/ready": read tcp 10.132.0.2:52202->10.132.0.34:15021: read: connection reset by peer
19:59:08 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:08 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:11 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Unhealthy	Startup probe failed: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused
19:59:11 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	ProbeError	Startup probe error: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused body:
19:59:12	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	Started	Container started
19:59:12	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	Pulled	Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 14.651s (14.651s including waiting). Image size: 1089502687 bytes.
19:59:12	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-hcbgc	Created	Container created
19:59:20	opendatahub	maas-controller-8f6976955-8fvl7_532abbbe-74c5-424f-ba6b-63e6176f5895	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-8f6976955-8fvl7_532abbbe-74c5-424f-ba6b-63e6176f5895 became leader
19:59:41	kuadrant-system	multus	authorino-6f6944bcdd-l8dlh	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
19:59:41	kuadrant-system	replicaset-controller	authorino-d6cffbf65	SuccessfulCreate	Created pod: authorino-d6cffbf65-vwchz
19:59:41	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulDelete	Deleted pod: authorino-6f6944bcdd-l8dlh
19:59:41	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulCreate	Created pod: authorino-6f6944bcdd-l8dlh
19:59:41	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6f6944bcdd from 1 to 0
19:59:41	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-d6cffbf65 from 0 to 1
19:59:41	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6f6944bcdd from 0 to 1
19:59:42	kuadrant-system	kubelet	authorino-d6cffbf65-vwchz	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:59:42	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-d6cffbf65 from 1 to 0
19:59:42	kuadrant-system	kubelet	authorino-6f6944bcdd-l8dlh	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:59:42	kuadrant-system	kubelet	authorino-67fb44bc55-2x7gn	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:59:42	kuadrant-system	replicaset-controller	authorino-d6cffbf65	SuccessfulDelete	Deleted pod: authorino-d6cffbf65-vwchz
19:59:42	kuadrant-system	multus	authorino-d6cffbf65-vwchz	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
19:59:42	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-67fb44bc55 from 0 to 1
19:59:42	kuadrant-system	multus	authorino-67fb44bc55-2x7gn	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
19:59:42	kuadrant-system	replicaset-controller	authorino-67fb44bc55	SuccessfulCreate	Created pod: authorino-67fb44bc55-2x7gn
19:59:43	kuadrant-system	kubelet	authorino-6f6944bcdd-l8dlh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.592s (1.592s including waiting). Image size: 204096810 bytes.
19:59:43	kuadrant-system	kubelet	authorino-6f6944bcdd-l8dlh	Started	Container started
19:59:43	kuadrant-system	kubelet	authorino-6f6944bcdd-l8dlh	Created	Container created
19:59:44	kuadrant-system	kubelet	authorino-d6cffbf65-vwchz	Started	Container started
19:59:44	kuadrant-system	kubelet	authorino-67fb44bc55-2x7gn	Created	Container created
19:59:44	kuadrant-system	kubelet	authorino-67fb44bc55-2x7gn	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.436s (1.436s including waiting). Image size: 204096810 bytes.
19:59:44	kuadrant-system	kubelet	authorino-6f6944bcdd-l8dlh	Killing	Stopping container authorino
19:59:44	kuadrant-system	kubelet	authorino-d6cffbf65-vwchz	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.905s (1.905s including waiting). Image size: 204096810 bytes.
19:59:44	kuadrant-system	kubelet	authorino-d6cffbf65-vwchz	Created	Container created
19:59:44	kuadrant-system	kubelet	authorino-67fb44bc55-2x7gn	Started	Container started
19:59:45	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulDelete	Deleted pod: authorino-56c6c9ddd5-snmh8
19:59:45	kuadrant-system	kubelet	authorino-56c6c9ddd5-snmh8	Killing	Stopping container authorino
19:59:45	kuadrant-system	kubelet	authorino-d6cffbf65-vwchz	Killing	Stopping container authorino
19:59:45	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-56c6c9ddd5 from 1 to 0
19:59:53 (x5)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:53 (x5)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:17	kuadrant-system	replicaset-controller	authorino-7d8bb9f59f	SuccessfulCreate	Created pod: authorino-7d8bb9f59f-xv9s5
20:00:18	kuadrant-system	kubelet	authorino-7d8bb9f59f-xv9s5	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:00:18	kuadrant-system	multus	authorino-7d8bb9f59f-xv9s5	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
20:00:21	kuadrant-system	kubelet	authorino-7d8bb9f59f-xv9s5	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.426s (2.426s including waiting). Image size: 204096810 bytes.
20:00:21	kuadrant-system	kubelet	authorino-7d8bb9f59f-xv9s5	Created	Container created
20:00:21	kuadrant-system	kubelet	authorino-7d8bb9f59f-xv9s5	Started	Container started
20:00:22	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1
20:00:22	kuadrant-system	replicaset-controller	authorino-67fb44bc55	SuccessfulDelete	Deleted pod: authorino-67fb44bc55-2x7gn
20:00:22	kuadrant-system	kubelet	authorino-67fb44bc55-2x7gn	Killing	Stopping container authorino
20:00:23	opendatahub	replicaset-controller	llmisvc-controller-manager-6466c554d8	SuccessfulCreate	Created pod: llmisvc-controller-manager-6466c554d8-7b7lk
20:00:23	opendatahub	multus	llmisvc-controller-manager-6466c554d8-7b7lk	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
20:00:23	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-7b7lk	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
20:00:24	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-7b7lk	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 715ms (715ms including waiting). Image size: 235100981 bytes.
20:00:24	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-7b7lk	Started	Container started
20:00:24	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-7b7lk	Created	Container created
20:00:56	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulDelete	Deleted pod: llmisvc-controller-manager-869d5dcc5d-b786t
20:00:56	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Killing	Stopping container manager
20:00:56	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0
20:01:01	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-b786t	Unhealthy	Readiness probe failed: Get "http://10.134.0.34:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:01:13	opendatahub	llmisvc-controller-manager-6466c554d8-7b7lk_b56b7aa2-ea99-47f5-925d-544d947100ea	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-6466c554d8-7b7lk_b56b7aa2-ea99-47f5-925d-544d947100ea became leader
20:01:16	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-5d8dddb676 from 0 to 1
20:01:16	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulCreate	Created pod: ai-gateway-controller-5d8dddb676-xsc87
20:01:17	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-xsc87	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
20:01:17	opendatahub	multus	ai-gateway-controller-5d8dddb676-xsc87	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
20:01:23	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-xsc87	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 6.034s (6.034s including waiting). Image size: 184136103 bytes.
20:01:23	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-xsc87	Created	Container created
20:01:23	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-xsc87	Started	Container started
20:01:24	opendatahub	ai-gateway-controller-5d8dddb676-xsc87_5185c7dd-0fa1-4f96-89c6-940b49137565	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-5d8dddb676-xsc87_5185c7dd-0fa1-4f96-89c6-940b49137565 became leader
20:01:27	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
20:01:27 (x6)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:01:27	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-7bb9489676 from 0 to 1
20:01:27	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-wpsp2
20:01:27	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	SuccessfulCreate	Created pod: payload-processing-7bb9489676-qcnxp
20:01:27 (x7)	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	FailedCreate	Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:01:28	openshift-ingress	multus	payload-pre-processing-7b98bb678c-wpsp2	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
20:01:28	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-wpsp2	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:01:28	openshift-ingress	multus	payload-processing-7bb9489676-qcnxp	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
20:01:28	openshift-ingress	kubelet	payload-processing-7bb9489676-qcnxp	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:01:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-wpsp2	Started	Container started
20:01:31	openshift-ingress	kubelet	payload-processing-7bb9489676-qcnxp	Created	Container created
20:01:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-wpsp2	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.735s (2.735s including waiting). Image size: 125353963 bytes.
20:01:31	openshift-ingress	kubelet	payload-processing-7bb9489676-qcnxp	Started	Container started
20:01:31	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-wpsp2	Created	Container created
20:01:31	openshift-ingress	kubelet	payload-processing-7bb9489676-qcnxp	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.597s (2.597s including waiting). Image size: 125353963 bytes.
20:01:34	opendatahub	replicaset-controller	ai-gateway-controller-598949b94	SuccessfulCreate	Created pod: ai-gateway-controller-598949b94-lwknj
20:01:34	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-598949b94 from 0 to 1
20:01:35	opendatahub	multus	ai-gateway-controller-598949b94-lwknj	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
20:01:35	opendatahub	kubelet	ai-gateway-controller-598949b94-lwknj	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
20:01:36	opendatahub	kubelet	ai-gateway-controller-598949b94-lwknj	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 839ms (839ms including waiting). Image size: 184136103 bytes.
20:01:36	opendatahub	kubelet	ai-gateway-controller-598949b94-lwknj	Started	Container started
20:01:36	opendatahub	kubelet	ai-gateway-controller-598949b94-lwknj	Created	Container created
20:01:47	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled down replica set ai-gateway-controller-5d8dddb676 from 1 to 0
20:01:47	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulDelete	Deleted pod: ai-gateway-controller-5d8dddb676-xsc87
20:01:47	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-xsc87	Killing	Stopping container manager
20:01:57	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Service llm/e2e-distinct-2-simulated-kserve-workload-svc
20:01:57	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Deployment llm/e2e-distinct-2-simulated-kserve
20:01:57	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
20:01:57	llm	deployment-controller	e2e-distinct-2-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-2-simulated-kserve-cbd4f5586 from 0 to 1
20:01:57	llm	replicaset-controller	e2e-distinct-2-simulated-kserve-cbd4f5586	SuccessfulCreate	Created pod: e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6
20:01:57	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-2-simulated-kserve-route
20:01:58	llm	multus	e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
20:01:58	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:01:58	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine
20:01:58	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine-default
20:01:58	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-2-simulated-kserve-workload-svc
20:01:59	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-2-simulated-prometheus-scraping
20:02:00	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	Created	Container created
20:02:00	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.368s (2.368s including waiting). Image size: 98346788 bytes.
20:02:00	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-nxbc6	Started	Container started
20:02:03	opendatahub	ai-gateway-controller-598949b94-lwknj_d88bd590-05ff-4ed6-b924-7b83b4f183ba	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-598949b94-lwknj_d88bd590-05ff-4ed6-b924-7b83b4f183ba became leader
20:02:04	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
20:02:04	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-simulated-kserve-route
20:02:04	llm	replicaset-controller	e2e-distinct-simulated-kserve-7b998b79d9	SuccessfulCreate	Created pod: e2e-distinct-simulated-kserve-7b998b79d9-2v7cp
20:02:04	llm	deployment-controller	e2e-distinct-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-simulated-kserve-7b998b79d9 from 0 to 1
20:02:04	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Deployment llm/e2e-distinct-simulated-kserve
20:02:04	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Service llm/e2e-distinct-simulated-kserve-workload-svc
20:02:05	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-simulated-prometheus-scraping
20:02:05	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine
20:02:05	llm	multus	e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
20:02:05	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:05	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-simulated-kserve-workload-svc
20:02:05	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine-default
20:02:06	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 602ms (602ms including waiting). Image size: 98346788 bytes.
20:02:06	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	Started	Container started
20:02:06	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-2v7cp	Created	Container created
20:02:07	llm	replicaset-controller	e2e-embedding-simulated-kserve-67bf594cd6	SuccessfulCreate	Created pod: e2e-embedding-simulated-kserve-67bf594cd6-ttrwc
20:02:07	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
20:02:07	llm	deployment-controller	e2e-embedding-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-embedding-simulated-kserve-67bf594cd6 from 0 to 1
20:02:07	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Service llm/e2e-embedding-simulated-kserve-workload-svc
20:02:07	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Deployment llm/e2e-embedding-simulated-kserve
20:02:08	llm	multus	e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
20:02:08	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.HTTPRoute llm/e2e-embedding-simulated-kserve-route
20:02:08	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.DestinationRule llm/e2e-embedding-simulated-kserve-workload-svc
20:02:08	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:08	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine
20:02:08	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine-default
20:02:09	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.NetworkPolicy llm/e2e-embedding-simulated-prometheus-scraping
20:02:10	llm	deployment-controller	e2e-trlp-test-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-trlp-test-simulated-kserve-69b588897d from 0 to 1
20:02:10	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.583s (1.583s including waiting). Image size: 98346788 bytes.
20:02:10	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	Created	Container created
20:02:10	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-ttrwc	Started	Container started
20:02:10	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Deployment llm/e2e-trlp-test-simulated-kserve
20:02:10	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
20:02:10	llm	replicaset-controller	e2e-trlp-test-simulated-kserve-69b588897d	SuccessfulCreate	Created pod: e2e-trlp-test-simulated-kserve-69b588897d-rfs4d
20:02:11	llm	multus	e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
20:02:11	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine-default
20:02:11	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine
20:02:11	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:11	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.DestinationRule llm/e2e-trlp-test-simulated-kserve-workload-svc
20:02:11	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Service llm/e2e-trlp-test-simulated-kserve-workload-svc
20:02:11	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.HTTPRoute llm/e2e-trlp-test-simulated-kserve-route
20:02:12	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	Started	Container started
20:02:12	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	Created	Container created
20:02:12	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-rfs4d	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 588ms (588ms including waiting). Image size: 98346788 bytes.
20:02:12	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.NetworkPolicy llm/e2e-trlp-test-simulated-prometheus-scraping
20:02:14	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
20:02:14	llm	replicaset-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc	SuccessfulCreate	Created pod: e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5
20:02:14	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Service llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
20:02:14	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Deployment llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve
20:02:14	llm	deployment-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc from 0 to 1
20:02:15	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-route
20:02:15	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/eab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine-default
20:02:15	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
20:02:15	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/e2e-unconab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine
20:02:15	llm	multus	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
20:02:15	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:16	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	Created	Container created
20:02:16	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 604ms (604ms including waiting). Image size: 98346788 bytes.
20:02:16	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c7zc5	Started	Container started
20:02:16	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-prometheus-scraping
20:02:17	llm	deployment-controller	echo-simulator-kserve	ScalingReplicaSet	Scaled up replica set echo-simulator-kserve-64b6f8f96c from 0 to 1
20:02:17	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Secret llm/echo-simulator-kserve-self-signed-certs
20:02:17	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Deployment llm/echo-simulator-kserve
20:02:18	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.HTTPRoute llm/echo-simulator-kserve-route
20:02:18	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6rx96	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0"
20:02:18	llm	replicaset-controller	echo-simulator-kserve-64b6f8f96c	SuccessfulCreate	Created pod: echo-simulator-kserve-64b6f8f96c-6rx96
20:02:18	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Service llm/echo-simulator-kserve-workload-svc
20:02:18	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.DestinationRule llm/echo-simulator-kserve-workload-svc
20:02:18	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine-default
20:02:18	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine
20:02:18	llm	multus	echo-simulator-kserve-64b6f8f96c-6rx96	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
20:02:19	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.NetworkPolicy llm/echo-simulator-prometheus-scraping
20:02:21	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Service llm/facebook-opt-125m-simulated-kserve-workload-svc
20:02:21	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6rx96	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0" in 2.648s (2.648s including waiting). Image size: 78208878 bytes.
20:02:21	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6rx96	Created	Container created
20:02:21	llm	kubelet	echo-simulator-kserve-64b6f8f96c-6rx96	Started	Container started
20:02:21	llm	replicaset-controller	facebook-opt-125m-simulated-kserve-84b4ff656c	SuccessfulCreate	Created pod: facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t
20:02:21	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
20:02:21	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Deployment llm/facebook-opt-125m-simulated-kserve
20:02:21	llm	deployment-controller	facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set facebook-opt-125m-simulated-kserve-84b4ff656c from 0 to 1
20:02:22	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:22	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/facebook-opt-125m-simulated-kserve-route
20:02:22	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine
20:02:22	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine-default
20:02:22	llm	multus	facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
20:02:22	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/facebook-opt-125m-simulated-kserve-workload-svc
20:02:23	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/facebook-opt-125m-simulated-prometheus-scraping
20:02:23	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 616ms (616ms including waiting). Image size: 98346788 bytes.
20:02:23	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	Started	Container started
20:02:23	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-n5w2t	Created	Container created
20:02:25	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
20:02:25	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.DestinationRule llm/premium-simulated-simulated-premium-kserve-workload-svc
20:02:25	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Deployment llm/premium-simulated-simulated-premium-kserve
20:02:25	llm	deployment-controller	premium-simulated-simulated-premium-kserve	ScalingReplicaSet	Scaled up replica set premium-simulated-simulated-premium-kserve-5f5c4b459f from 0 to 1
20:02:25	llm	replicaset-controller	premium-simulated-simulated-premium-kserve-5f5c4b459f	SuccessfulCreate	Created pod: premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb
20:02:25	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Service llm/premium-simulated-simulated-premium-kserve-workload-svc
20:02:25	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.HTTPRoute llm/premium-simulated-simulated-premium-kserve-route
20:02:26	llm	multus	premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
20:02:26	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/p75d89895ad19b4ac62612d08bfe7f574-kserve-llmisvc-engine-default
20:02:26	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/premium-simulated-simulated-premium-kserve-llmisvc-engine
20:02:26	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:26	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.NetworkPolicy llm/premium-simulated-simulated-premium-prometheus-scraping
20:02:26	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 636ms (636ms including waiting). Image size: 98346788 bytes.
20:02:26	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	Created	Container created
20:02:26	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-x58jb	Started	Container started
20:02:28 (x2)	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Updated	Updated v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
20:02:29	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-2-simulated] is Ready
20:02:31	llm	LLMInferenceServiceController	e2e-distinct-simulated	Updated	Updated v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
20:02:33	llm	LLMInferenceServiceController	e2e-distinct-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-simulated] is Ready
20:02:34	llm	LLMInferenceServiceController	e2e-embedding-simulated	Updated	Updated v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
20:02:36	llm	LLMInferenceServiceController	e2e-embedding-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-embedding-simulated] is Ready
20:02:40	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Updated	Updated v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
20:02:40	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-59f957fdd7 from 1 to 2
20:02:40	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 2; reason: cpu resource utilization (percentage of request) above target
20:02:40	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-6hkkd
20:02:41	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
20:02:41	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
20:02:42	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-trlp-test-simulated] is Ready
20:02:43	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
20:02:44	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-unconfigured-facebook-opt-125m-simulated] is Ready
20:02:46	llm	LLMInferenceServiceController	echo-simulator	Updated	Updated v1.Secret llm/echo-simulator-kserve-self-signed-certs
20:02:48	llm	LLMInferenceServiceController	echo-simulator	LLMInferenceServiceReady	LLMInferenceService [echo-simulator] is Ready
20:02:49	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
20:02:50	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [facebook-opt-125m-simulated] is Ready
20:02:51	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Started	Container started
20:02:51	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.685s (10.685s including waiting). Image size: 183555333 bytes.
20:02:51	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Created	Container created
20:02:52	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Updated	Updated v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
20:02:53	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	LLMInferenceServiceReady	LLMInferenceService [premium-simulated-simulated-premium] is Ready
20:02:55	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
20:02:55	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-59f957fdd7 from 2 to 4
20:02:55	kuadrant-system	replicaset-controller	authorino-d8886d8d8	SuccessfulCreate	Created pod: authorino-d8886d8d8-cfg9x
20:02:55	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
20:02:55	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-xj5qx
20:02:55	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-xtsh2
20:02:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Started	Container started
20:02:56	kuadrant-system	kubelet	authorino-d8886d8d8-cfg9x	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:02:56	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
20:02:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:02:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Created	Container created
20:02:56	kuadrant-system	multus	authorino-d8886d8d8-cfg9x	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
20:02:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
20:02:58 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	ProbeError	Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused body:
20:02:58	kuadrant-system	kubelet	authorino-d8886d8d8-cfg9x	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.695s (1.695s including waiting). Image size: 204096810 bytes.
20:02:58	kuadrant-system	kubelet	authorino-d8886d8d8-cfg9x	Created	Container created
20:02:58	kuadrant-system	kubelet	authorino-d8886d8d8-cfg9x	Started	Container started
20:02:58 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Unhealthy	Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused
20:02:59	kuadrant-system	replicaset-controller	authorino-7d8bb9f59f	SuccessfulDelete	Deleted pod: authorino-7d8bb9f59f-xv9s5
20:02:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Created	Container created
20:02:59 (x4)	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	(combined from similar events): Scaled down replica set authorino-7d8bb9f59f from 1 to 0
20:02:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.889s (3.889s including waiting). Image size: 183555333 bytes.
20:02:59	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Started	Container started
20:03:00	kuadrant-system	kubelet	authorino-7d8bb9f59f-xv9s5	Killing	Stopping container authorino
20:03:02 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	ProbeError	Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused body:
20:03:02 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Unhealthy	Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused
20:03:06 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	ProbeError	Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused body:
20:03:06 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xtsh2	Unhealthy	Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused
20:03:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-lgvxj
20:03:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-tqlvx
20:03:10	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-59f957fdd7 from 4 to 6
20:03:10	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 6; reason:
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Started	Container started
20:03:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Started	Container started
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Created	Container created
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Created	Container created
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:03:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
20:03:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:03:17 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Unhealthy	Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused
20:03:17 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	ProbeError	Startup probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused body:
20:03:17 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	ProbeError	Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused body:
20:03:17 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Unhealthy	Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.DestinationRule llm/grid-inv-b695bb2e-model-kserve-workload-svc
20:03:32	llm	multus	grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
20:03:32	llm	replicaset-controller	grid-inv-b695bb2e-model-kserve-6dc694749	SuccessfulCreate	Created pod: grid-inv-b695bb2e-model-kserve-6dc694749-t5m85
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.Service llm/grid-inv-b695bb2e-model-kserve-workload-svc
20:03:32	llm	deployment-controller	grid-inv-b695bb2e-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-b695bb2e-model-kserve-6dc694749 from 0 to 1
20:03:32	llm	kubelet	grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.HTTPRoute llm/grid-inv-b695bb2e-model-kserve-route
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.PodMonitor llm/grid-inv-b695bb2e-model-kserve-llmisvc-engine-default
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.Deployment llm/grid-inv-b695bb2e-model-kserve
20:03:32	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.Secret llm/grid-inv-b695bb2e-model-kserve-self-signed-certs
20:03:33	llm	kubelet	grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	Started	Container started
20:03:33	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.NetworkPolicy llm/grid-inv-b695bb2e-model-prometheus-scraping
20:03:33	llm	kubelet	grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	Created	Container created
20:03:33	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Created	Created v1.PodMonitor llm/grid-inv-b695bb2e-model-kserve-llmisvc-engine
20:03:36	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	Updated	Updated v1.Secret llm/grid-inv-b695bb2e-model-kserve-self-signed-certs
20:03:55	llm	replicaset-controller	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847	SuccessfulCreate	Created pod: grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx
20:03:55	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.Service llm/grid-inv-b695bb2e-unmatched-kserve-workload-svc
20:03:55	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.Secret llm/grid-inv-b695bb2e-unmatched-kserve-self-signed-certs
20:03:55	llm	deployment-controller	grid-inv-b695bb2e-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-b695bb2e-unmatched-kserve-cd9dc9847 from 0 to 1
20:03:55	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.Deployment llm/grid-inv-b695bb2e-unmatched-kserve
20:03:56	llm	kubelet	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	Created	Container created
20:03:56	llm	kubelet	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	Started	Container started
20:03:56	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.PodMonitor llm/grid-inv-b695bb2e-unmatched-kserve-llmisvc-engine
20:03:56	llm	kubelet	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:03:56	llm	multus	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
20:03:56	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-b695bb2e-unmatched-kserve-route
20:03:56	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.DestinationRule llm/grid-inv-b695bb2e-unmatched-kserve-workload-svc
20:03:56	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.PodMonitor llm/grid-inv-b695bb2e-unmatched-kserve-llmisvc-engine-default
20:03:57	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-b695bb2e-unmatched-prometheus-scraping
20:03:59	llm	LLMInferenceServiceController	grid-inv-b695bb2e-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-b695bb2e-model] is Ready
20:04:01	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Error	Reconciliation failed: failed to reconcile networking: failed to reconcile HTTP routes: failed to reconcile HTTPRoute llm/grid-inv-b695bb2e-unmatched-kserve-route: failed to get defaults for v1.HTTPRoute llm/grid-inv-b695bb2e-unmatched-kserve-route: Operation cannot be fulfilled on httproutes.gateway.networking.k8s.io "grid-inv-b695bb2e-unmatched-kserve-route": the object has been modified; please apply your changes to the latest version and try again
20:04:01	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	Updated	Updated v1.Secret llm/grid-inv-b695bb2e-unmatched-kserve-self-signed-certs
20:04:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-gvnng
20:04:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-2x8vt
20:04:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-59f957fdd7-kndtt
20:04:10	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-59f957fdd7	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-59f957fdd7-n79ks
20:04:10	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-59f957fdd7 from 6 to 10
20:04:10	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	Created	Container created
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	Started	Container started
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	Started	Container started
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-kndtt	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
20:04:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Started	Container started
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	Started	Container started
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	Created	Container created
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-gvnng	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
20:04:11	openshift-ingress	multus	maas-default-gateway-openshift-default-59f957fdd7-n79ks	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Created	Container created
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:11	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	Created	Container created
20:04:19 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	ProbeError	Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
20:04:19 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	ProbeError	Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused body:
20:04:19 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Unhealthy	Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused
20:04:19 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	Unhealthy	Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused
20:04:19 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	Unhealthy	Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused
20:04:19 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	ProbeError	Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused body:
20:04:19 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	ProbeError	Startup probe error: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused body:
20:04:19 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	Unhealthy	Startup probe failed: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused
20:04:21	llm	LLMInferenceServiceController	grid-inv-b695bb2e-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-b695bb2e-unmatched] is Ready
20:05:25	llm	kubelet	grid-inv-b695bb2e-model-kserve-6dc694749-t5m85	Killing	Stopping container main
20:05:40	llm	kubelet	grid-inv-b695bb2e-unmatched-kserve-cd9dc9847-bnjhx	Killing	Stopping container main
20:05:59	llm	replicaset-controller	grid-inv-93130f4e-model-kserve-954f6d98c	SuccessfulCreate	Created pod: grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8
20:05:59	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.DestinationRule llm/grid-inv-93130f4e-model-kserve-workload-svc
20:05:59	llm	deployment-controller	grid-inv-93130f4e-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-93130f4e-model-kserve-954f6d98c from 0 to 1
20:05:59	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.Secret llm/grid-inv-93130f4e-model-kserve-self-signed-certs
20:05:59	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.Deployment llm/grid-inv-93130f4e-model-kserve
20:05:59	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.Service llm/grid-inv-93130f4e-model-kserve-workload-svc
20:05:59	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.HTTPRoute llm/grid-inv-93130f4e-model-kserve-route
20:06:00	llm	kubelet	grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	Started	Container started
20:06:00	llm	multus	grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
20:06:00	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.PodMonitor llm/grid-inv-93130f4e-model-kserve-llmisvc-engine
20:06:00	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.NetworkPolicy llm/grid-inv-93130f4e-model-prometheus-scraping
20:06:00	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Created	Created v1.PodMonitor llm/grid-inv-93130f4e-model-kserve-llmisvc-engine-default
20:06:00	llm	kubelet	grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	Created	Container created
20:06:00	llm	kubelet	grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:06:03	llm	multus	grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
20:06:03	llm	kubelet	grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.DestinationRule llm/grid-inv-93130f4e-unmatched-kserve-workload-svc
20:06:03	llm	replicaset-controller	grid-inv-93130f4e-unmatched-kserve-85d4f456df	SuccessfulCreate	Created pod: grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg
20:06:03	llm	deployment-controller	grid-inv-93130f4e-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-93130f4e-unmatched-kserve-85d4f456df from 0 to 1
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.Secret llm/grid-inv-93130f4e-unmatched-kserve-self-signed-certs
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.Deployment llm/grid-inv-93130f4e-unmatched-kserve
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.Service llm/grid-inv-93130f4e-unmatched-kserve-workload-svc
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-93130f4e-unmatched-kserve-route
20:06:03	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.PodMonitor llm/grid-inv-93130f4e-unmatched-kserve-llmisvc-engine-default
20:06:04	llm	kubelet	grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	Started	Container started
20:06:04	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-93130f4e-unmatched-prometheus-scraping
20:06:04	llm	kubelet	grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	Created	Container created
20:06:04	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Created	Created v1.PodMonitor llm/grid-inv-93130f4e-unmatched-kserve-llmisvc-engine
20:06:08	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	Updated	Updated v1.Secret llm/grid-inv-93130f4e-model-kserve-self-signed-certs
20:06:24	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	Updated	Updated v1.Secret llm/grid-inv-93130f4e-unmatched-kserve-self-signed-certs
20:06:26	llm	LLMInferenceServiceController	grid-inv-93130f4e-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-93130f4e-unmatched] is Ready
20:06:28	llm	LLMInferenceServiceController	grid-inv-93130f4e-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-93130f4e-model] is Ready
20:06:33	opendatahub	kubelet	test-curl-9041-2c82f4	Pulling	Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"
20:06:33	opendatahub	multus	test-curl-9041-2c82f4	AddedInterface	Add eth0 [10.132.0.64/23] from ovn-kubernetes
20:06:35	opendatahub	kubelet	test-curl-9041-2c82f4	Created	Container created
20:06:35	opendatahub	kubelet	test-curl-9041-2c82f4	Started	Container started
20:06:35	opendatahub	kubelet	test-curl-9041-2c82f4	Pulled	Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.353s (2.354s including waiting). Image size: 106100986 bytes.
20:06:38	opendatahub	kubelet	test-curl-9041-2c82f4	Killing	Stopping container test-curl-9041-2c82f4
20:06:39	opendatahub	kubelet	test-curl-9041-a678b6	Started	Container started
20:06:39	opendatahub	multus	test-curl-9041-a678b6	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
20:06:39	opendatahub	kubelet	test-curl-9041-a678b6	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
20:06:39	opendatahub	kubelet	test-curl-9041-a678b6	Created	Container created
20:06:42	opendatahub	kubelet	test-curl-9041-a678b6	Killing	Stopping container test-curl-9041-a678b6
20:06:43	opendatahub	multus	test-curl-9041-19b575	AddedInterface	Add eth0 [10.132.0.66/23] from ovn-kubernetes
20:06:43	opendatahub	kubelet	test-curl-9041-19b575	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
20:06:43	opendatahub	kubelet	test-curl-9041-19b575	Created	Container created
20:06:43	opendatahub	kubelet	test-curl-9041-19b575	Started	Container started
20:06:45	opendatahub	kubelet	test-curl-9041-19b575	Killing	Stopping container test-curl-9041-19b575
20:06:45	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-e56f6e	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-e56f6e
20:06:46	openshift-ingress	multus	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	AddedInterface	Add eth0 [10.132.0.67/23] from ovn-kubernetes
20:06:46	openshift-ingress	replicaset-controller	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5	SuccessfulCreate	Created pod: e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8
20:06:46	openshift-ingress	deployment-controller	e2e-shared-a-main-e56f6e-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-e56f6e-openshift-default-656776d9d5 from 0 to 1
20:06:53	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-e56f6e	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59 from 0 to 1
20:06:53	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-e56f6e	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-e56f6e-6c854875f4 from 0 to 1
20:06:53 (x7)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-e56f6e: serviceaccount "payload-processing-e2e-shared-a-main-e56f6e" not found
20:06:53 (x6)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-e56f6e: serviceaccount "payload-processing-e2e-shared-a-main-e56f6e" not found
20:06:54	openshift-ingress	multus	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	AddedInterface	Add eth0 [10.132.0.69/23] from ovn-kubernetes
20:06:54	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l
20:06:54	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494
20:06:55	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	AddedInterface	Add eth0 [10.132.0.68/23] from ovn-kubernetes
20:06:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:06:55	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:06:57	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.768s (2.768s including waiting). Image size: 125353963 bytes.
20:06:58	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Created	Container created
20:06:58	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Started	Container started
20:06:58	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Created	Container created
20:06:58	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Started	Container started
20:06:58	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.96s (2.96s including waiting). Image size: 125353963 bytes.
20:07:15 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Created	Container created
20:07:15 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:07:15 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Started	Container started
20:07:16	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-12288c	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-12288c
20:07:16	openshift-ingress	replicaset-controller	e2e-shared-b-main-12288c-openshift-default-754fff74df	SuccessfulCreate	Created pod: e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r
20:07:16	openshift-ingress	deployment-controller	e2e-shared-b-main-12288c-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-12288c-openshift-default-754fff74df from 0 to 1
20:07:17	openshift-ingress	multus	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	AddedInterface	Add eth0 [10.132.0.70/23] from ovn-kubernetes
20:07:21 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	ProbeError	Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused body:
20:07:21 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Unhealthy	Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused
20:07:23	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-12288c	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-12288c-55f4d7c574 from 0 to 1
20:07:23	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-12288c	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c9 from 0 to 1
20:07:24	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-12288c-55f4d7c574	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w
20:07:24 (x7)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-12288c-55f4d7c574	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-12288c-55f4d7c574-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-12288c: serviceaccount "payload-processing-e2e-shared-b-main-12288c" not found
20:07:24	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
20:07:24	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	AddedInterface	Add eth0 [10.132.0.71/23] from ovn-kubernetes
20:07:24	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c9	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g
20:07:24 (x6)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c9	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-12288c: serviceaccount "payload-processing-e2e-shared-b-main-12288c" not found
20:07:24	openshift-ingress	multus	payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
20:07:25	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	Started	Container started
20:07:25	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	Created	Container created
20:07:25	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	Started	Container started
20:07:25	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
20:07:25	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	Created	Container created
20:07:31 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e56f6e-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:07:31 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e56f6e-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:07:46	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e56f6e-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:46	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-e56f6e-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:07:49	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Unhealthy	Readiness probe failed: Get "http://10.132.0.70:15021/healthz/ready": read tcp 10.132.0.2:51608->10.132.0.70:15021: read: connection reset by peer
20:07:49	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	ProbeError	Readiness probe error: Get "http://10.132.0.70:15021/healthz/ready": read tcp 10.132.0.2:51608->10.132.0.70:15021: read: connection reset by peer body:
20:07:49 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Created	Container created
20:07:49 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:07:50 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Started	Container started
20:07:54 (x4)	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	ProbeError	Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused body:
20:07:54 (x4)	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Unhealthy	Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused
20:08:17 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-12288c-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:17 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-12288c-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:57	opendatahub	kubelet	test-curl-9041-0ce8b3	Pulling	Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"
20:08:57	opendatahub	multus	test-curl-9041-0ce8b3	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
20:08:58	opendatahub	kubelet	test-curl-9041-0ce8b3	Created	Container created
20:08:58	opendatahub	kubelet	test-curl-9041-0ce8b3	Pulled	Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 1.026s (1.026s including waiting). Image size: 106100986 bytes.
20:08:58	opendatahub	kubelet	test-curl-9041-0ce8b3	Started	Container started
20:09:01	opendatahub	kubelet	test-curl-9041-0ce8b3	Killing	Stopping container test-curl-9041-0ce8b3
20:09:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	ProbeError	Readiness probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:09:31	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-6hkkd	Unhealthy	Readiness probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:09:50 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	ProbeError	Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:09:50 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-tqlvx	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:10:15	llm	kubelet	grid-inv-93130f4e-model-kserve-954f6d98c-gfvs8	Killing	Stopping container main
20:10:21	llm	kubelet	grid-inv-93130f4e-unmatched-kserve-85d4f456df-qqvrg	Killing	Stopping container main
20:10:28 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	ProbeError	Readiness probe error: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:10:28 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-q6cx2	Unhealthy	Readiness probe failed: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:10:42	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.Secret llm/grid-inv-cf634b53-model-kserve-self-signed-certs
20:10:42	llm	replicaset-controller	grid-inv-cf634b53-model-kserve-58488b84d4	SuccessfulCreate	Created pod: grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk
20:10:42	llm	deployment-controller	grid-inv-cf634b53-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-cf634b53-model-kserve-58488b84d4 from 0 to 1
20:10:42	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.Deployment llm/grid-inv-cf634b53-model-kserve
20:10:42	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.Service llm/grid-inv-cf634b53-model-kserve-workload-svc
20:10:43	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.DestinationRule llm/grid-inv-cf634b53-model-kserve-workload-svc
20:10:43	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.HTTPRoute llm/grid-inv-cf634b53-model-kserve-route
20:10:43	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.PodMonitor llm/grid-inv-cf634b53-model-kserve-llmisvc-engine-default
20:10:43	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.PodMonitor llm/grid-inv-cf634b53-model-kserve-llmisvc-engine
20:10:43	llm	multus	grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
20:10:43	llm	kubelet	grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	Started	Container started
20:10:43	llm	kubelet	grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	Created	Container created
20:10:43	llm	kubelet	grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:10:44	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Created	Created v1.NetworkPolicy llm/grid-inv-cf634b53-model-prometheus-scraping
20:10:46	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.Secret llm/grid-inv-cf634b53-unmatched-kserve-self-signed-certs
20:10:46	llm	replicaset-controller	grid-inv-cf634b53-unmatched-kserve-544d9f7f54	SuccessfulCreate	Created pod: grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z
20:10:46	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-cf634b53-unmatched-kserve-route
20:10:46	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.Service llm/grid-inv-cf634b53-unmatched-kserve-workload-svc
20:10:46	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.Deployment llm/grid-inv-cf634b53-unmatched-kserve
20:10:46	llm	deployment-controller	grid-inv-cf634b53-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-cf634b53-unmatched-kserve-544d9f7f54 from 0 to 1
20:10:47	llm	multus	grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
20:10:47	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.PodMonitor llm/grid-inv-cf634b53-unmatched-kserve-llmisvc-engine-default
20:10:47	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.DestinationRule llm/grid-inv-cf634b53-unmatched-kserve-workload-svc
20:10:47	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.PodMonitor llm/grid-inv-cf634b53-unmatched-kserve-llmisvc-engine
20:10:47	llm	kubelet	grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	Started	Container started
20:10:47	llm	kubelet	grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	Created	Container created
20:10:47	llm	kubelet	grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:10:48	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-cf634b53-unmatched-prometheus-scraping
20:10:50 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	ProbeError	Readiness probe error: Get "http://10.133.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:10:50 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-lgvxj	Unhealthy	Readiness probe failed: Get "http://10.133.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:10:50 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	ProbeError	Readiness probe error: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:10:50 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-xj5qx	Unhealthy	Readiness probe failed: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:10:51 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	Unhealthy	Readiness probe failed: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:10:51 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-2x8vt	ProbeError	Readiness probe error: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:10:52	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	Updated	Updated v1.Secret llm/grid-inv-cf634b53-model-kserve-self-signed-certs
20:11:11	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	Updated	Updated v1.Secret llm/grid-inv-cf634b53-unmatched-kserve-self-signed-certs
20:11:12	llm	LLMInferenceServiceController	grid-inv-cf634b53-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-cf634b53-unmatched] is Ready
20:11:14	llm	LLMInferenceServiceController	grid-inv-cf634b53-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-cf634b53-model] is Ready
20:12:21 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	Unhealthy	Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:12:21 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-kndtt	ProbeError	Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:12:21 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	Unhealthy	Readiness probe failed: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:12:21 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:12:21 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-gvnng	ProbeError	Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:12:21 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-59f957fdd7-n79ks	ProbeError	Readiness probe error: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:12:54	llm	kubelet	grid-inv-cf634b53-model-kserve-58488b84d4-m4qvk	Killing	Stopping container main
20:13:01	llm	kubelet	grid-inv-cf634b53-unmatched-kserve-544d9f7f54-qdf5z	Killing	Stopping container main
20:13:38	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-e56f6e-6c854875f4-s4494	Killing	Stopping container payload-processing
20:13:38	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-e56f6e-754b95b59-df49l	Killing	Stopping container payload-pre-processing
20:13:46	openshift-ingress	kubelet	e2e-shared-a-main-e56f6e-openshift-default-656776d9d5-5j8b8	Killing	Stopping container istio-proxy
20:13:53	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-12288c-55f4d7c574-8k45w	Killing	Stopping container payload-processing
20:13:53	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-12288c-67c58f56c96ms6g	Killing	Stopping container payload-pre-processing
20:14:06	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Killing	Stopping container istio-proxy
20:14:11	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:14:11	openshift-ingress	kubelet	e2e-shared-b-main-12288c-openshift-default-754fff74df-9jc5r	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body: