Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:56:47	openshift-ingress		openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	Scheduled	Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd to ip-10-0-130-189.us-east-2.compute.internal
20:03:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-5xhnp	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5xhnp to ip-10-0-135-240.us-east-2.compute.internal
19:57:44	cert-manager		cert-manager-597b95cb84-5tj7q	Scheduled	Successfully assigned cert-manager/cert-manager-597b95cb84-5tj7q to ip-10-0-135-240.us-east-2.compute.internal
19:43:40	openshift-ingress		router-default-5bd87cbd45-9b8cf	FailedScheduling	no nodes available to schedule pods
19:49:50	openshift-ingress		router-default-5bd87cbd45-9b8cf	Scheduled	Successfully assigned openshift-ingress/router-default-5bd87cbd45-9b8cf to ip-10-0-130-189.us-east-2.compute.internal
19:58:03	kuadrant-system		kuadrant-operator-controller-manager-678d87766f-xt9sx	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xt9sx to ip-10-0-135-240.us-east-2.compute.internal
20:03:00	llm		premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	Scheduled	Successfully assigned llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm to ip-10-0-128-228.us-east-2.compute.internal
20:05:26	opendatahub		test-curl-8726-5c62b6	Scheduled	Successfully assigned opendatahub/test-curl-8726-5c62b6 to ip-10-0-128-228.us-east-2.compute.internal
20:02:48	llm		e2e-distinct-simulated-kserve-7b998b79d9-5qx79	Scheduled	Successfully assigned llm/e2e-distinct-simulated-kserve-7b998b79d9-5qx79 to ip-10-0-128-228.us-east-2.compute.internal
19:57:34	cert-manager		cert-manager-cainjector-5c57bbd4fc-lpg8m	Scheduled	Successfully assigned cert-manager/cert-manager-cainjector-5c57bbd4fc-lpg8m to ip-10-0-135-240.us-east-2.compute.internal
20:05:09	llm		grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	Scheduled	Successfully assigned llm/grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7 to ip-10-0-128-228.us-east-2.compute.internal
19:58:53	openshift-ingress		istiod-openshift-gateway-7ddd9cfdc8-z7cw2	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7cw2 to ip-10-0-128-228.us-east-2.compute.internal
20:08:00	llm		grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	Scheduled	Successfully assigned llm/grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d to ip-10-0-128-228.us-east-2.compute.internal
19:57:52	openshift-ingress		kube-auth-proxy-d655f64d9-2sv5x	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-d655f64d9-2sv5x to ip-10-0-135-240.us-east-2.compute.internal
19:57:52	openshift-ingress		kube-auth-proxy-d655f64d9-wnbhd	Scheduled	Successfully assigned openshift-ingress/kube-auth-proxy-d655f64d9-wnbhd to ip-10-0-128-228.us-east-2.compute.internal
20:02:50	llm		e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	Scheduled	Successfully assigned llm/e2e-embedding-simulated-kserve-67bf594cd6-cjsx6 to ip-10-0-135-240.us-east-2.compute.internal
20:07:58	llm		grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	Scheduled	Successfully assigned llm/grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8 to ip-10-0-130-189.us-east-2.compute.internal
20:04:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-88lvc	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-88lvc to ip-10-0-135-240.us-east-2.compute.internal
19:57:32	cert-manager		cert-manager-webhook-5dbbbf75cc-nhmj6	Scheduled	Successfully assigned cert-manager/cert-manager-webhook-5dbbbf75cc-nhmj6 to ip-10-0-135-240.us-east-2.compute.internal
19:57:50	kuadrant-system		authorino-operator-7b7bb4bc77-mgchw	Scheduled	Successfully assigned kuadrant-system/authorino-operator-7b7bb4bc77-mgchw to ip-10-0-130-189.us-east-2.compute.internal
20:03:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-8vwd6	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8vwd6 to ip-10-0-128-228.us-east-2.compute.internal
20:05:31	opendatahub		test-curl-8726-77e414	Scheduled	Successfully assigned opendatahub/test-curl-8726-77e414 to ip-10-0-128-228.us-east-2.compute.internal
20:05:55	openshift-ingress		payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc to ip-10-0-128-228.us-east-2.compute.internal
19:58:10	openshift-jobset-operator		jobset-controller-manager-7974c69c9-z7zkd	Scheduled	Successfully assigned openshift-jobset-operator/jobset-controller-manager-7974c69c9-z7zkd to ip-10-0-130-189.us-east-2.compute.internal
20:05:41	openshift-ingress		payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Scheduled	Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl to ip-10-0-135-240.us-east-2.compute.internal
20:05:13	llm		grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Scheduled	Successfully assigned llm/grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf to ip-10-0-130-189.us-east-2.compute.internal
20:02:15	openshift-ingress		payload-processing-7bb9489676-7mbkz	Scheduled	Successfully assigned openshift-ingress/payload-processing-7bb9489676-7mbkz to ip-10-0-135-240.us-east-2.compute.internal
20:05:55	openshift-ingress		payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n to ip-10-0-128-228.us-east-2.compute.internal
20:05:41	openshift-ingress		payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g to ip-10-0-128-228.us-east-2.compute.internal
20:02:59	llm		facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	Scheduled	Successfully assigned llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5 to ip-10-0-128-228.us-east-2.compute.internal
20:01:19	kuadrant-system		authorino-555b588c49-rg2lv	Scheduled	Successfully assigned kuadrant-system/authorino-555b588c49-rg2lv to ip-10-0-135-240.us-east-2.compute.internal
20:02:15	openshift-ingress		payload-pre-processing-7b98bb678c-zl76j	Scheduled	Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-zl76j to ip-10-0-135-240.us-east-2.compute.internal
20:06:42	opendatahub		test-curl-8726-4fa8a8	Scheduled	Successfully assigned opendatahub/test-curl-8726-4fa8a8 to ip-10-0-128-228.us-east-2.compute.internal
19:58:52	opendatahub		odh-model-controller-769875db56-lm88t	Scheduled	Successfully assigned opendatahub/odh-model-controller-769875db56-lm88t to ip-10-0-130-189.us-east-2.compute.internal
19:58:52	opendatahub		model-serving-api-75bd89fdfc-svr6c	Scheduled	Successfully assigned opendatahub/model-serving-api-75bd89fdfc-svr6c to ip-10-0-135-240.us-east-2.compute.internal
20:03:53	llm		grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	Scheduled	Successfully assigned llm/grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25 to ip-10-0-128-228.us-east-2.compute.internal
19:59:47	opendatahub		maas-controller-655c897cd5-pf8cd	Scheduled	Successfully assigned opendatahub/maas-controller-655c897cd5-pf8cd to ip-10-0-128-228.us-east-2.compute.internal
19:59:48	opendatahub		maas-controller-5655cc7547-q2mfq	Scheduled	Successfully assigned opendatahub/maas-controller-5655cc7547-q2mfq to ip-10-0-128-228.us-east-2.compute.internal
19:57:46	kuadrant-system		kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Scheduled	Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-g7zkh to ip-10-0-130-189.us-east-2.compute.internal
19:57:59	kuadrant-system		authorino-56c6c9ddd5-fngsn	Scheduled	Successfully assigned kuadrant-system/authorino-56c6c9ddd5-fngsn to ip-10-0-135-240.us-east-2.compute.internal
19:58:52	opendatahub		llmisvc-controller-manager-869d5dcc5d-scdxv	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-scdxv to ip-10-0-130-189.us-east-2.compute.internal
19:57:55	kuadrant-system		kuadrant-console-plugin-66c4fb575d-xgcmn	Scheduled	Successfully assigned kuadrant-system/kuadrant-console-plugin-66c4fb575d-xgcmn to ip-10-0-135-240.us-east-2.compute.internal
20:01:21	opendatahub		llmisvc-controller-manager-6466c554d8-d6fbl	Scheduled	Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-d6fbl to ip-10-0-130-189.us-east-2.compute.internal
19:57:33	openshift-jobset-operator		jobset-operator-674cd59dc5-2zdpn	Scheduled	Successfully assigned openshift-jobset-operator/jobset-operator-674cd59dc5-2zdpn to ip-10-0-135-240.us-east-2.compute.internal
20:02:57	llm		echo-simulator-kserve-64b6f8f96c-kgwfb	Scheduled	Successfully assigned llm/echo-simulator-kserve-64b6f8f96c-kgwfb to ip-10-0-135-240.us-east-2.compute.internal
19:57:12	opendatahub-operator-system		opendatahub-operator-controller-manager-5cf78c489b-krbbk	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-5cf78c489b-krbbk to ip-10-0-135-240.us-east-2.compute.internal
20:02:55	llm		e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	Scheduled	Successfully assigned llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw to ip-10-0-135-240.us-east-2.compute.internal
19:57:57	kuadrant-system		limitador-limitador-74dfcfb894-5ch2c	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-74dfcfb894-5ch2c to ip-10-0-135-240.us-east-2.compute.internal
19:57:58	kuadrant-system		authorino-57f7f8659f-5w46d	Scheduled	Successfully assigned kuadrant-system/authorino-57f7f8659f-5w46d to ip-10-0-130-189.us-east-2.compute.internal
19:57:58	kuadrant-system		limitador-limitador-7c9b87d574-w6hvg	Scheduled	Successfully assigned kuadrant-system/limitador-limitador-7c9b87d574-w6hvg to ip-10-0-128-228.us-east-2.compute.internal
20:10:39	ai-tenants		e2e-shared-b-main-dbce1a	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-b-main-dbce1a: MaasTenantConfig is not available to restore revocation operands
20:03:33	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-9hc8b	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hc8b to ip-10-0-130-189.us-east-2.compute.internal
19:59:53	opendatahub		usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	Scheduled	Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r9d4h to ip-10-0-128-228.us-east-2.compute.internal
19:57:57	opendatahub		kserve-module-controller-manager-7566cf5fc8-gxp8x	Scheduled	Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-gxp8x to ip-10-0-130-189.us-east-2.compute.internal
20:03:33	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-wrztq	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wrztq to ip-10-0-128-228.us-east-2.compute.internal
20:10:25	ai-tenants		e2e-shared-a-main-18ecf5	APIKeyCleanupSkipped	Skipping API-key cleanup for AITenant ai-tenants/e2e-shared-a-main-18ecf5: MaasTenantConfig is not available to restore revocation operands
20:04:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-wgkhr	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wgkhr to ip-10-0-130-189.us-east-2.compute.internal
20:03:13	kuadrant-system		authorino-66ccd8f8b-vgm8p	Scheduled	Successfully assigned kuadrant-system/authorino-66ccd8f8b-vgm8p to ip-10-0-135-240.us-east-2.compute.internal
20:03:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-vft4f	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vft4f to ip-10-0-135-240.us-east-2.compute.internal
20:03:57	llm		grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	Scheduled	Successfully assigned llm/grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk to ip-10-0-128-228.us-east-2.compute.internal
20:02:52	llm		e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	Scheduled	Successfully assigned llm/e2e-trlp-test-simulated-kserve-69b588897d-mg8s7 to ip-10-0-128-228.us-east-2.compute.internal
20:03:33	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-kxp6g	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kxp6g to ip-10-0-135-240.us-east-2.compute.internal
20:04:18	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-f6z8p	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6z8p to ip-10-0-128-228.us-east-2.compute.internal
19:58:52	opendatahub		kserve-controller-manager-776cc66d47-hzzhx	Scheduled	Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-hzzhx to ip-10-0-128-228.us-east-2.compute.internal
20:00:34	kuadrant-system		authorino-6889cbd8c4-pbgz6	Scheduled	Successfully assigned kuadrant-system/authorino-6889cbd8c4-pbgz6 to ip-10-0-135-240.us-east-2.compute.internal
20:02:25	opendatahub		ai-gateway-controller-76f789867f-ftf5k	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-76f789867f-ftf5k to ip-10-0-135-240.us-east-2.compute.internal
19:57:12	opendatahub-operator-system		opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-5cf78c489b-4fc7k to ip-10-0-130-189.us-east-2.compute.internal
20:02:45	llm		e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	Scheduled	Successfully assigned llm/e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks to ip-10-0-135-240.us-east-2.compute.internal
20:02:10	opendatahub		ai-gateway-controller-5d8dddb676-w4rzz	Scheduled	Successfully assigned opendatahub/ai-gateway-controller-5d8dddb676-w4rzz to ip-10-0-135-240.us-east-2.compute.internal
19:57:38	kuadrant-system		limitador-operator-controller-manager-5778bb4449-6bkrk	Scheduled	Successfully assigned kuadrant-system/limitador-operator-controller-manager-5778bb4449-6bkrk to ip-10-0-130-189.us-east-2.compute.internal
19:57:52	openshift-ingress		data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8	Scheduled	Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8 to ip-10-0-130-189.us-east-2.compute.internal
19:57:12	opendatahub-operator-system		opendatahub-operator-controller-manager-5cf78c489b-krhsz	Scheduled	Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-5cf78c489b-krhsz to ip-10-0-128-228.us-east-2.compute.internal
20:05:36	openshift-ingress		e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Scheduled	Successfully assigned openshift-ingress/e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s to ip-10-0-128-228.us-east-2.compute.internal
20:00:34	kuadrant-system		authorino-6f6944bcdd-zhjjl	Scheduled	Successfully assigned kuadrant-system/authorino-6f6944bcdd-zhjjl to ip-10-0-130-189.us-east-2.compute.internal
20:05:50	openshift-ingress		e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Scheduled	Successfully assigned openshift-ingress/e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh to ip-10-0-135-240.us-east-2.compute.internal
19:56:43	openshift-ingress		istiod-openshift-gateway-7876bfcb74-w4fzz	Scheduled	Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4fzz to ip-10-0-128-228.us-east-2.compute.internal
20:00:34	kuadrant-system		authorino-94899989-mcjbn	Scheduled	Successfully assigned kuadrant-system/authorino-94899989-mcjbn to ip-10-0-130-189.us-east-2.compute.internal
20:05:33	opendatahub		test-curl-8726-952975	Scheduled	Successfully assigned opendatahub/test-curl-8726-952975 to ip-10-0-128-228.us-east-2.compute.internal
19:43:38	openshift-ingress		router-default-5bd87cbd45-9b8cf	FailedScheduling	no nodes available to schedule pods
19:57:54	kuadrant-system		dns-operator-controller-manager-8557cbdb84-khsdp	Scheduled	Successfully assigned kuadrant-system/dns-operator-controller-manager-8557cbdb84-khsdp to ip-10-0-130-189.us-east-2.compute.internal
19:56:47	openshift-ingress		maas-default-gateway-openshift-default-67bf986f99-9jg2l	Scheduled	Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9jg2l to ip-10-0-130-189.us-east-2.compute.internal
19:43:04	openshift-ingress	service-controller	router-default	UnAvailableLoadBalancer	There are no available nodes for LoadBalancer
19:43:04	openshift-ingress	service-controller	router-default	EnsuringLoadBalancer	Ensuring load balancer
19:43:04	openshift-ingress	deployment-controller	router-default	ScalingReplicaSet	Scaled up replica set router-default-5bd87cbd45 from 0 to 1
19:43:07	openshift-ingress	service-controller	router-default	EnsuredLoadBalancer	Ensured load balancer
19:43:14	openshift-ingress	replicaset-controller	router-default-5bd87cbd45	FailedCreate	Error creating: pods "router-default-5bd87cbd45-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range
19:43:38	openshift-ingress	replicaset-controller	router-default-5bd87cbd45	SuccessfulCreate	Created pod: router-default-5bd87cbd45-9b8cf
19:49:31 (x3)	openshift-ingress	service-controller	router-default	UpdatedLoadBalancer	Updated load balancer with new hosts
19:49:58 (x5)	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
19:49:58 (x5)	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	FailedMount	MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
19:50:05 (x3)	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	FailedMount	MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
19:50:08	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	Started	Container started
19:50:08	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	Created	Container created
19:50:08	openshift-ingress	kubelet	router-default-5bd87cbd45-9b8cf	Pulled	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod
19:50:08	openshift-ingress	multus	router-default-5bd87cbd45-9b8cf	AddedInterface	Add eth0 [10.132.0.15/23] from ovn-kubernetes
19:56:41	openshift-ingress	gateway_labeler_controller	maas-default-gateway	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway
19:56:43	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1
19:56:43	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulCreate	Created pod: istiod-openshift-gateway-7876bfcb74-w4fzz
19:56:43	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-w4fzz	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"
19:56:43	openshift-ingress	multus	istiod-openshift-gateway-7876bfcb74-w4fzz	AddedInterface	Add eth0 [10.133.0.19/23] from ovn-kubernetes
19:56:46	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-w4fzz	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.985s (2.985s including waiting). Image size: 185476477 bytes.
19:56:47	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-w4fzz	Created	Container created
19:56:47	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuringLoadBalancer	Ensuring load balancer
19:56:47	openshift-ingress	deployment-controller	openshift-ai-inference-openshift-ai-inference	ScalingReplicaSet	Scaled up replica set openshift-ai-inference-openshift-ai-inference-67dd9d5b47 from 0 to 1
19:56:47	openshift-ingress	replicaset-controller	openshift-ai-inference-openshift-ai-inference-67dd9d5b47	SuccessfulCreate	Created pod: openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd
19:56:47	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-w4fzz	Started	Container started
19:56:47	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 0 to 1
19:56:47	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-9jg2l
19:56:48	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-9jg2l	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
19:56:48	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
19:56:48	openshift-ingress	multus	openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
19:56:48	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
19:56:50	openshift-ingress	service-controller	openshift-ai-inference-openshift-ai-inference	EnsuredLoadBalancer	Ensured load balancer
19:56:56	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 7.972s (7.972s including waiting). Image size: 183555333 bytes.
19:56:56	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	Created	Container created
19:56:56	openshift-ingress	kubelet	openshift-ai-inference-openshift-ai-inference-67dd9d5b47-ps6kd	Started	Container started
19:56:56	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.042s (8.042s including waiting). Image size: 183555333 bytes.
19:57:10 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsNotMet	one or more requirements couldn't be found
19:57:10 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	RequirementsUnknown	requirements not yet checked
19:57:11	opendatahub-operator-system	deployment-controller	opendatahub-operator-controller-manager	ScalingReplicaSet	Scaled up replica set opendatahub-operator-controller-manager-5cf78c489b from 0 to 3
19:57:12	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-5cf78c489b	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-5cf78c489b-krbbk
19:57:12	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-5cf78c489b	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-5cf78c489b-krhsz
19:57:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-5cf78c489b-krhsz	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
19:57:12	opendatahub-operator-system	replicaset-controller	opendatahub-operator-controller-manager-5cf78c489b	SuccessfulCreate	Created pod: opendatahub-operator-controller-manager-5cf78c489b-4fc7k
19:57:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
19:57:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:12	opendatahub-operator-system	multus	opendatahub-operator-controller-manager-5cf78c489b-krbbk	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
19:57:12	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:13	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	Webhook install failed: conversionWebhook not ready
19:57:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Started	Container started
19:57:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Created	Container created
19:57:15	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 2.929s (2.929s including waiting). Image size: 273709574 bytes.
19:57:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 385ms (385ms including waiting). Image size: 273709574 bytes.
19:57:16	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	NeedsReinstall	calculated deployment install is bad
19:57:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Started	Container started
19:57:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Created	Container created
19:57:16	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krhsz	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:16 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	AllRequirementsMet	all requirements found, attempting install
19:57:16	opendatahub-operator-system	opendatahub-operator-controller-manager-5cf78c489b-krhsz_3264545c-20f2-4396-9cf2-6495633624ef	07ed84f7.opendatahub.io	LeaderElection	opendatahub-operator-controller-manager-5cf78c489b-krhsz_3264545c-20f2-4396-9cf2-6495633624ef became leader
19:57:17 (x4)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	waiting for install components to report healthy
19:57:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:17 (x2)	opendatahub-operator-system	operator-lifecycle-manager	install-d96vn	AppliedWithWarnings	1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1
19:57:17 (x2)	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:18 (x2)	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallWaiting	installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability.
19:57:19	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 7.14s (7.14s including waiting). Image size: 273709574 bytes.
19:57:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Created	Container created
19:57:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Created	Container created
19:57:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Started	Container started
19:57:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Started	Container started
19:57:20	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 7.354s (7.354s including waiting). Image size: 273709574 bytes.
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Started	Container started
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Created	Container created
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 441ms (441ms including waiting). Image size: 273709574 bytes.
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Created	Container created
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-4fc7k	Pulled	Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 391ms (391ms including waiting). Image size: 273709574 bytes.
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Started	Container started
19:57:21	opendatahub-operator-system	kubelet	opendatahub-operator-controller-manager-5cf78c489b-krbbk	Pulling	Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
19:57:29	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsUnknown	requirements not yet checked
19:57:29	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	RequirementsNotMet	one or more requirements couldn't be found
19:57:32	cert-manager	deployment-controller	cert-manager-webhook	ScalingReplicaSet	Scaled up replica set cert-manager-webhook-5dbbbf75cc from 0 to 1
19:57:32	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	AllRequirementsMet	all requirements found, attempting install
19:57:32	cert-manager	multus	cert-manager-webhook-5dbbbf75cc-nhmj6	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:57:32	cert-manager	deployment-controller	cert-manager-cainjector	ScalingReplicaSet	Scaled up replica set cert-manager-cainjector-5c57bbd4fc from 0 to 1
19:57:32	cert-manager	replicaset-controller	cert-manager-webhook-5dbbbf75cc	SuccessfulCreate	Created pod: cert-manager-webhook-5dbbbf75cc-nhmj6
19:57:33	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:33	openshift-jobset-operator	multus	jobset-operator-674cd59dc5-2zdpn	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
19:57:33 (x9)	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	FailedCreate	Error creating: pods "cert-manager-cainjector-5c57bbd4fc-" is forbidden: error looking up service account cert-manager/cert-manager-cainjector: serviceaccount "cert-manager-cainjector" not found
19:57:33	openshift-ingress	horizontal-pod-autoscaler	openshift-ai-inference-openshift-ai-inference	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:33	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-nhmj6	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
19:57:33 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	waiting for install components to report healthy
19:57:33	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallWaiting	installing: waiting for deployment jobset-operator to become ready: deployment "jobset-operator" not available: Deployment does not have minimum availability.
19:57:33	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
19:57:33	openshift-jobset-operator	deployment-controller	jobset-operator	ScalingReplicaSet	Scaled up replica set jobset-operator-674cd59dc5 from 0 to 1
19:57:33	openshift-jobset-operator	replicaset-controller	jobset-operator-674cd59dc5	SuccessfulCreate	Created pod: jobset-operator-674cd59dc5-2zdpn
19:57:33	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
19:57:33	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-2zdpn	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e"
19:57:34	opendatahub-operator-system	operator-lifecycle-manager	opendatahub-operator.v3.6.0-ea.2	InstallSucceeded	install strategy completed with no errors
19:57:34	cert-manager	replicaset-controller	cert-manager-cainjector-5c57bbd4fc	SuccessfulCreate	Created pod: cert-manager-cainjector-5c57bbd4fc-lpg8m
19:57:35	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-lpg8m	Pulling	Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"
19:57:35	cert-manager	multus	cert-manager-cainjector-5c57bbd4fc-lpg8m	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
19:57:37	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-nhmj6	Created	Container created
19:57:37	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-2zdpn	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e" in 3.564s (3.564s including waiting). Image size: 224874686 bytes.
19:57:37	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-nhmj6	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 4.816s (4.816s including waiting). Image size: 309650901 bytes.
19:57:37	cert-manager	kubelet	cert-manager-webhook-5dbbbf75cc-nhmj6	Started	Container started
19:57:37	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-2zdpn	Started	Container started
19:57:37	openshift-jobset-operator	kubelet	jobset-operator-674cd59dc5-2zdpn	Created	Container created
19:57:37	openshift-jobset-operator	openshift-jobset-operator	openshift-jobset-operator-lock	LeaderElection	jobset-operator-674cd59dc5-2zdpn_1be86752-d1ef-4bc5-81b9-1a49280bb4b8 became leader
19:57:38	kuadrant-system	replicaset-controller	limitador-operator-controller-manager-5778bb4449	SuccessfulCreate	Created pod: limitador-operator-controller-manager-5778bb4449-6bkrk
19:57:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
19:57:38	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
19:57:38 (x2)	openshift-jobset-operator	operator-lifecycle-manager	jobset-operator.v1.0.1	InstallSucceeded	install strategy completed with no errors
19:57:38	kuadrant-system	deployment-controller	limitador-operator-controller-manager	ScalingReplicaSet	Scaled up replica set limitador-operator-controller-manager-5778bb4449 from 0 to 1
19:57:38 (x2)	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallWaiting	installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.
19:57:38	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-lpg8m	Pulled	Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 3.216s (3.216s including waiting). Image size: 309650901 bytes.
19:57:38	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
19:57:38	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-lpg8m	Started	Container started
19:57:38	cert-manager	kubelet	cert-manager-cainjector-5c57bbd4fc-lpg8m	Created	Container created
19:57:39	cert-manager	deployment-controller	cert-manager	ScalingReplicaSet	Scaled up replica set cert-manager-597b95cb84 from 0 to 1
19:57:39	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
19:57:39	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-6bkrk	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e"
19:57:39	kuadrant-system	multus	limitador-operator-controller-manager-5778bb4449-6bkrk	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
19:57:41 (x10)	cert-manager	replicaset-controller	cert-manager-597b95cb84	FailedCreate	Error creating: pods "cert-manager-597b95cb84-" is forbidden: error looking up service account cert-manager/cert-manager: serviceaccount "cert-manager" not found
19:57:42	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-6bkrk	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e" in 3.598s (3.598s including waiting). Image size: 177967631 bytes.
19:57:43	kuadrant-system	limitador-operator-controller-manager-5778bb4449-6bkrk_1b266f49-bd40-40d3-84eb-351060a49f5b	3745a16e.kuadrant.io	LeaderElection	limitador-operator-controller-manager-5778bb4449-6bkrk_1b266f49-bd40-40d3-84eb-351060a49f5b became leader
19:57:43	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-6bkrk	Created	Container created
19:57:43	kuadrant-system	kubelet	limitador-operator-controller-manager-5778bb4449-6bkrk	Started	Container started
19:57:44	cert-manager	kubelet	cert-manager-597b95cb84-5tj7q	Pulled	Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod
19:57:44	cert-manager	multus	cert-manager-597b95cb84-5tj7q	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
19:57:44	cert-manager	replicaset-controller	cert-manager-597b95cb84	SuccessfulCreate	Created pod: cert-manager-597b95cb84-5tj7q
19:57:45	cert-manager	kubelet	cert-manager-597b95cb84-5tj7q	Created	Container created
19:57:45	cert-manager	kubelet	cert-manager-597b95cb84-5tj7q	Started	Container started
19:57:46	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-cf8469d7b	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-cf8469d7b-g7zkh
19:57:46	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsUnknown	requirements not yet checked
19:57:46	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.
19:57:46	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
19:57:46	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	RequirementsNotMet	one or more requirements couldn't be found
19:57:46	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-cf8469d7b from 0 to 1
19:57:47	kuadrant-system	multus	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
19:57:47	kuadrant-system	kubelet	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"
19:57:50	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	AllRequirementsMet	all requirements found, attempting install
19:57:50	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallWaiting	installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.
19:57:50	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsNotMet	one or more requirements couldn't be found
19:57:50	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
19:57:50	kuadrant-system	replicaset-controller	authorino-operator-7b7bb4bc77	SuccessfulCreate	Created pod: authorino-operator-7b7bb4bc77-mgchw
19:57:50	kuadrant-system	deployment-controller	authorino-operator	ScalingReplicaSet	Scaled up replica set authorino-operator-7b7bb4bc77 from 0 to 1
19:57:50	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	RequirementsUnknown	requirements not yet checked
19:57:51	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-mgchw	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589"
19:57:51	kuadrant-system	multus	authorino-operator-7b7bb4bc77-mgchw	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
19:57:52	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceCreated	Created Service/jobset-webhook-service -n openshift-jobset-operator because it was missing
19:57:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 4.917s (4.917s including waiting). Image size: 424244516 bytes.
19:57:52	openshift-ingress	replicaset-controller	kube-auth-proxy-d655f64d9	SuccessfulCreate	Created pod: kube-auth-proxy-d655f64d9-2sv5x
19:57:52	openshift-ingress	deployment-controller	kube-auth-proxy	ScalingReplicaSet	Scaled up replica set kube-auth-proxy-d655f64d9 from 0 to 2
19:57:52	openshift-ingress	replicaset-controller	data-science-gateway-data-science-gateway-class-6f89d9f6cc	SuccessfulCreate	Created pod: data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8
19:57:52	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceAccountCreated	Created ServiceAccount/jobset-controller-manager -n openshift-jobset-operator because it was missing
19:57:52	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	IssuerCreated	Created Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator because it was missing
19:57:52	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-serving-cert -n openshift-jobset-operator because it was missing
19:57:52	openshift-ingress	deployment-controller	data-science-gateway-data-science-gateway-class	ScalingReplicaSet	Scaled up replica set data-science-gateway-data-science-gateway-class-6f89d9f6cc from 0 to 1
19:57:52	kuadrant-system	kuadrant-operator-controller-manager-cf8469d7b-g7zkh_1a1dacf5-13aa-439c-9976-8493016522a2	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-cf8469d7b-g7zkh_1a1dacf5-13aa-439c-9976-8493016522a2 became leader
19:57:52	openshift-ingress	replicaset-controller	kube-auth-proxy-d655f64d9	SuccessfulCreate	Created pod: kube-auth-proxy-d655f64d9-wnbhd
19:57:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Started	Container started
19:57:52	kuadrant-system	kubelet	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Created	Container created
19:57:53	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CertificateCreated	Created Certificate.cert-manager.io/jobset-metrics-cert -n openshift-jobset-operator because it was missing
19:57:53	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8	Started	Container started
19:57:53	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-2sv5x	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
19:57:53	openshift-ingress	multus	kube-auth-proxy-d655f64d9-2sv5x	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
19:57:53	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-mgchw	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589" in 1.941s (1.941s including waiting). Image size: 167738465 bytes.
19:57:53	kuadrant-system	authorino-operator-7b7bb4bc77-mgchw_f12cc6fb-c4b7-4d89-8383-acd3e3d67d9c	aac3a15d.authorino.kuadrant.io	LeaderElection	authorino-operator-7b7bb4bc77-mgchw_f12cc6fb-c4b7-4d89-8383-acd3e3d67d9c became leader
19:57:53	openshift-ingress	multus	data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
19:57:53	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
19:57:53	openshift-ingress	kubelet	data-science-gateway-data-science-gateway-class-6f89d9f6cc7qsq8	Created	Container created
19:57:53	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-mgchw	Created	Container created
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-manager-role because it was missing
19:57:53	kuadrant-system	kubelet	authorino-operator-7b7bb4bc77-mgchw	Started	Container started
19:57:53	openshift-ingress	multus	kube-auth-proxy-d655f64d9-wnbhd	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-metrics-reader because it was missing
19:57:53	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-wnbhd	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleCreated	Created ClusterRole.rbac.authorization.k8s.io/jobset-proxy-role because it was missing
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-manager-rolebinding because it was missing
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding because it was missing
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ClusterRoleBindingCreated	Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding because it was missing
19:57:53	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-leader-election-role -n openshift-jobset-operator because it was missing
19:57:54	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	AllRequirementsMet	all requirements found, attempting install
19:57:54	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding -n openshift-jobset-operator because it was missing
19:57:54	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-manager-secrets-role -n openshift-jobset-operator because it was missing
19:57:54	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallWaiting	installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.
19:57:54	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	ServiceCreated	Created Service/jobset-controller-manager-metrics-service -n openshift-jobset-operator because it was missing
19:57:54	openshift-jobset-operator	openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-manager-secrets-rolebinding -n openshift-jobset-operator because it was missing
19:57:54	kuadrant-system	operator-lifecycle-manager	limitador-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
19:57:54	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	waiting for install components to report healthy
19:57:54	kuadrant-system	deployment-controller	dns-operator-controller-manager	ScalingReplicaSet	Scaled up replica set dns-operator-controller-manager-8557cbdb84 from 0 to 1
19:57:54	kuadrant-system	replicaset-controller	dns-operator-controller-manager-8557cbdb84	SuccessfulCreate	Created pod: dns-operator-controller-manager-8557cbdb84-khsdp
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-wnbhd	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.524s (2.524s including waiting). Image size: 188904677 bytes.
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-2sv5x	Started	Container started
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-wnbhd	Created	Container created
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-2sv5x	Created	Container created
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-2sv5x	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.359s (2.359s including waiting). Image size: 188904677 bytes.
19:57:55	openshift-ingress	kubelet	kube-auth-proxy-d655f64d9-wnbhd	Started	Container started
19:57:55	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-khsdp	Pulling	Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a"
19:57:55	kuadrant-system	multus	dns-operator-controller-manager-8557cbdb84-khsdp	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
19:57:55	kuadrant-system	deployment-controller	kuadrant-console-plugin	ScalingReplicaSet	Scaled up replica set kuadrant-console-plugin-66c4fb575d from 0 to 1
19:57:55	kuadrant-system	replicaset-controller	kuadrant-console-plugin-66c4fb575d	SuccessfulCreate	Created pod: kuadrant-console-plugin-66c4fb575d-xgcmn
19:57:56	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-xgcmn	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044"
19:57:56	kuadrant-system	multus	kuadrant-console-plugin-66c4fb575d-xgcmn	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:57:57	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-74dfcfb894 from 0 to 1
19:57:57	opendatahub	deployment-controller	kserve-module-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1
19:57:57	opendatahub	replicaset-controller	kserve-module-controller-manager-7566cf5fc8	SuccessfulCreate	Created pod: kserve-module-controller-manager-7566cf5fc8-gxp8x
19:57:57	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulCreate	Created pod: limitador-limitador-74dfcfb894-5ch2c
19:57:58	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-khsdp	Created	Container created
19:57:58	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-w6hvg	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
19:57:58	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-57f7f8659f from 0 to 1
19:57:58	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-khsdp	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a" in 3.42s (3.42s including waiting). Image size: 190804149 bytes.
19:57:58	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulCreate	Created pod: authorino-57f7f8659f-5w46d
19:57:58	opendatahub	multus	kserve-module-controller-manager-7566cf5fc8-gxp8x	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
19:57:58	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled up replica set limitador-limitador-7c9b87d574 from 0 to 1
19:57:58	kuadrant-system	multus	limitador-limitador-74dfcfb894-5ch2c	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
19:57:58	kuadrant-system	replicaset-controller	limitador-limitador-7c9b87d574	SuccessfulCreate	Created pod: limitador-limitador-7c9b87d574-w6hvg
19:57:58	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-gxp8x	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189"
19:57:58	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-5ch2c	Pulling	Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"
19:57:58	kuadrant-system	multus	limitador-limitador-7c9b87d574-w6hvg	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
19:57:58	kuadrant-system	kubelet	dns-operator-controller-manager-8557cbdb84-khsdp	Started	Container started
19:57:59	kuadrant-system	kubelet	authorino-57f7f8659f-5w46d	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:57:59	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulCreate	Created pod: authorino-56c6c9ddd5-fngsn
19:57:59	kuadrant-system	multus	authorino-57f7f8659f-5w46d	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:57:59	kuadrant-system	multus	authorino-56c6c9ddd5-fngsn	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:57:59	kuadrant-system	kubelet	authorino-56c6c9ddd5-fngsn	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
19:57:59	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-56c6c9ddd5 from 0 to 1
19:57:59	kuadrant-system	dns-operator-controller-manager-8557cbdb84-khsdp_b8c59584-c952-498b-9276-6a4309f3b3e2	a3f98d6c.kuadrant.io	LeaderElection	dns-operator-controller-manager-8557cbdb84-khsdp_b8c59584-c952-498b-9276-6a4309f3b3e2 became leader
19:58:00	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-gxp8x	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.524s (2.524s including waiting). Image size: 195729000 bytes.
19:58:00	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-gxp8x	Created	Container created
19:58:00	opendatahub	kubelet	kserve-module-controller-manager-7566cf5fc8-gxp8x	Started	Container started
19:58:01	kuadrant-system	kubelet	authorino-56c6c9ddd5-fngsn	Started	Container started
19:58:01	kuadrant-system	kubelet	authorino-56c6c9ddd5-fngsn	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.205s (2.205s including waiting). Image size: 204096810 bytes.
19:58:01	kuadrant-system	kubelet	authorino-57f7f8659f-5w46d	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.039s (2.039s including waiting). Image size: 204096810 bytes.
19:58:01	kuadrant-system	kubelet	authorino-57f7f8659f-5w46d	Created	Container created
19:58:01	kuadrant-system	kubelet	authorino-57f7f8659f-5w46d	Started	Container started
19:58:01	kuadrant-system	kubelet	authorino-56c6c9ddd5-fngsn	Created	Container created
19:58:01	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-5ch2c	Started	Container started
19:58:01	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-5ch2c	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 2.856s (2.856s including waiting). Image size: 133379307 bytes.
19:58:01	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-5ch2c	Created	Container created
19:58:01	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-xgcmn	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044" in 5.407s (5.407s including waiting). Image size: 342538866 bytes.
19:58:01	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-xgcmn	Created	Container created
19:58:01	kuadrant-system	kubelet	kuadrant-console-plugin-66c4fb575d-xgcmn	Started	Container started
19:58:02	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-57f7f8659f from 1 to 0
19:58:02	kuadrant-system	replicaset-controller	authorino-57f7f8659f	SuccessfulDelete	Deleted pod: authorino-57f7f8659f-5w46d
19:58:03	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled up replica set kuadrant-operator-controller-manager-678d87766f from 0 to 1
19:58:03	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" waiting for 1 outdated replica(s) to be terminated
19:58:03	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-678d87766f	SuccessfulCreate	Created pod: kuadrant-operator-controller-manager-678d87766f-xt9sx
19:58:03	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-w6hvg	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 4.426s (4.426s including waiting). Image size: 133379307 bytes.
19:58:03	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-w6hvg	Created	Container created
19:58:03	kuadrant-system	kubelet	limitador-limitador-7c9b87d574-w6hvg	Started	Container started
19:58:03 (x2)	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	waiting for install components to report healthy
19:58:03	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallWaiting	installing: deployment changed old hash=6q1WY8Uae4JZmLF4M4Zj2lZF5tKn8wEFi4k5Ve, new hash=88W7Jv6nAXJm3xNFHpPpBPB6INrNa0R4IQ7iYv
19:58:04	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-xt9sx	Pulling	Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"
19:58:04	kuadrant-system	kubelet	authorino-57f7f8659f-5w46d	Killing	Stopping container authorino
19:58:04	kuadrant-system	multus	kuadrant-operator-controller-manager-678d87766f-xt9sx	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
19:58:04	kuadrant-system	operator-lifecycle-manager	authorino-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
19:58:07	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:07	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:08	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-xt9sx	Created	Container created
19:58:08	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-xt9sx	Started	Container started
19:58:08	kuadrant-system	kubelet	kuadrant-operator-controller-manager-678d87766f-xt9sx	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 4.489s (4.489s including waiting). Image size: 424244516 bytes.
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-venafi	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-serving-cert	Generated	Stored new private key in temporary Secret resource "jobset-serving-cert-4n5nn"
19:58:09	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-serving-cert	Issuing	The certificate has been successfully issued
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-vault	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-serving-cert-1	CertificateIssued	Certificate fetched from issuer successfully
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-acme	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-serving-cert	Requested	Created new CertificateRequest resource "jobset-serving-cert-1"
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-metrics-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-metrics-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-selfsigned	jobset-metrics-cert-1	CertificateIssued	Certificate fetched from issuer successfully
19:58:09	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-metrics-cert	Issuing	Issuing certificate as Secret does not exist
19:58:09	openshift-jobset-operator	cert-manager-certificates-key-manager	jobset-metrics-cert	Generated	Stored new private key in temporary Secret resource "jobset-metrics-cert-6lnng"
19:58:09	openshift-jobset-operator	cert-manager-certificates-request-manager	jobset-metrics-cert	Requested	Created new CertificateRequest resource "jobset-metrics-cert-1"
19:58:09	openshift-jobset-operator	cert-manager-certificates-issuing	jobset-metrics-cert	Issuing	The certificate has been successfully issued
19:58:09	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationCreated	Created ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it was missing
19:58:09	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationCreated	Created MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it was missing
19:58:09	openshift-jobset-operator	cert-manager-certificates-trigger	jobset-serving-cert	Issuing	Issuing certificate as Secret does not exist
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-issuer-ca	jobset-serving-cert-1	WaitingForApproval	Not signing CertificateRequest until it is Approved
19:58:09	openshift-jobset-operator	cert-manager-certificaterequests-approver	jobset-serving-cert-1	cert-manager.io	Certificate request has been approved by cert-manager.io
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleCreated	Created Role.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
19:58:10	openshift-jobset-operator	replicaset-controller	jobset-controller-manager-7974c69c9	SuccessfulCreate	Created pod: jobset-controller-manager-7974c69c9-z7zkd
19:58:10	openshift-jobset-operator	multus	jobset-controller-manager-7974c69c9-z7zkd	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	RoleBindingCreated	Created RoleBinding.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ServiceMonitorCreated	Created ServiceMonitor.monitoring.coreos.com/jobset-controller-manager-metrics-monitor -n openshift-jobset-operator because it was missing
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ConfigMapCreated	Created ConfigMap/jobset-manager-config -n openshift-jobset-operator because it was missing
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	DeploymentCreated	Created Deployment.apps/jobset-controller-manager -n openshift-jobset-operator because it was missing
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	ValidatingWebhookConfigurationUpdated	Updated ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it changed
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	MutatingWebhookConfigurationUpdated	Updated MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it changed
19:58:10	openshift-jobset-operator	kubelet	jobset-controller-manager-7974c69c9-z7zkd	Pulling	Pulling image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987"
19:58:10	openshift-jobset-operator	openshift-jobset-operator	jobset-operator	CustomResourceDefinitionCreated	Created CustomResourceDefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io because it was missing
19:58:10	openshift-jobset-operator	deployment-controller	jobset-controller-manager	ScalingReplicaSet	Scaled up replica set jobset-controller-manager-7974c69c9 from 0 to 1
19:58:10	kuadrant-system	operator-lifecycle-manager	dns-operator.v1.4.2	InstallSucceeded	install strategy completed with no errors
19:58:13	openshift-jobset-operator	kubelet	jobset-controller-manager-7974c69c9-z7zkd	Created	Container created
19:58:13	openshift-jobset-operator	kubelet	jobset-controller-manager-7974c69c9-z7zkd	Started	Container started
19:58:13	openshift-jobset-operator	kubelet	jobset-controller-manager-7974c69c9-z7zkd	Pulled	Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" in 2.931s (2.931s including waiting). Image size: 182184678 bytes.
19:58:14	kuadrant-system	deployment-controller	limitador-limitador	ScalingReplicaSet	Scaled down replica set limitador-limitador-74dfcfb894 from 1 to 0
19:58:14	kuadrant-system	kubelet	limitador-limitador-74dfcfb894-5ch2c	Killing	Stopping container limitador
19:58:14	kuadrant-system	replicaset-controller	limitador-limitador-74dfcfb894	SuccessfulDelete	Deleted pod: limitador-limitador-74dfcfb894-5ch2c
19:58:14	openshift-jobset-operator	jobset-controller-manager-7974c69c9-z7zkd_7d3fbbf2-a66e-4dfb-a669-bb595e79c7c6	6d4f6a47.jobset.x-k8s.io	LeaderElection	jobset-controller-manager-7974c69c9-z7zkd_7d3fbbf2-a66e-4dfb-a669-bb595e79c7c6 became leader
19:58:16	opendatahub	kserve-module-controller-manager-7566cf5fc8-gxp8x_63bcd7fc-3d6e-409c-b925-a530fde2f135	kserve-module-controller-manager	LeaderElection	kserve-module-controller-manager-7566cf5fc8-gxp8x_63bcd7fc-3d6e-409c-b925-a530fde2f135 became leader
19:58:20	kuadrant-system	kubelet	kuadrant-operator-controller-manager-cf8469d7b-g7zkh	Killing	Stopping container manager
19:58:20	kuadrant-system	deployment-controller	kuadrant-operator-controller-manager	ScalingReplicaSet	Scaled down replica set kuadrant-operator-controller-manager-cf8469d7b from 1 to 0
19:58:20	kuadrant-system	replicaset-controller	kuadrant-operator-controller-manager-cf8469d7b	SuccessfulDelete	Deleted pod: kuadrant-operator-controller-manager-cf8469d7b-g7zkh
19:58:21	kuadrant-system	operator-lifecycle-manager	rhcl-operator.v1.4.3	InstallSucceeded	install strategy completed with no errors
19:58:22	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:22	openshift-ingress	horizontal-pod-autoscaler	kube-auth-proxy	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:27 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:27 (x2)	openshift-ingress	horizontal-pod-autoscaler	data-science-gateway-data-science-gateway-class	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:37	kuadrant-system	kuadrant-operator-controller-manager-678d87766f-xt9sx_1a75e539-ff54-4755-a904-6168547026f3	f139389e.kuadrant.io	LeaderElection	kuadrant-operator-controller-manager-678d87766f-xt9sx_1a75e539-ff54-4755-a904-6168547026f3 became leader
19:58:52	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1
19:58:52	opendatahub	replicaset-controller	kserve-controller-manager-776cc66d47	SuccessfulCreate	Created pod: kserve-controller-manager-776cc66d47-hzzhx
19:58:52	opendatahub	replicaset-controller	odh-model-controller-769875db56	SuccessfulCreate	Created pod: odh-model-controller-769875db56-lm88t
19:58:52	opendatahub	replicaset-controller	model-serving-api-75bd89fdfc	SuccessfulCreate	Created pod: model-serving-api-75bd89fdfc-svr6c
19:58:52	opendatahub	deployment-controller	kserve-controller-manager	ScalingReplicaSet	Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1
19:58:52	opendatahub	deployment-controller	model-serving-api	ScalingReplicaSet	Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1
19:58:52	opendatahub	deployment-controller	odh-model-controller	ScalingReplicaSet	Scaled up replica set odh-model-controller-769875db56 from 0 to 1
19:58:52	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulCreate	Created pod: llmisvc-controller-manager-869d5dcc5d-scdxv
19:58:53	opendatahub	multus	model-serving-api-75bd89fdfc-svr6c	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:58:53	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-scdxv	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
19:58:53	opendatahub	kubelet	odh-model-controller-769875db56-lm88t	Pulling	Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775"
19:58:53	opendatahub	multus	odh-model-controller-769875db56-lm88t	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
19:58:53	opendatahub	multus	kserve-controller-manager-776cc66d47-hzzhx	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
19:58:53	opendatahub	kubelet	kserve-controller-manager-776cc66d47-hzzhx	Pulling	Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d"
19:58:53	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7ddd9cfdc8	SuccessfulCreate	Created pod: istiod-openshift-gateway-7ddd9cfdc8-z7cw2
19:58:53	opendatahub	multus	llmisvc-controller-manager-869d5dcc5d-scdxv	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
19:58:53	opendatahub	kubelet	model-serving-api-75bd89fdfc-svr6c	Pulling	Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8"
19:58:53	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1
19:58:54	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-z7cw2	Started	Container started
19:58:54	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-z7cw2	Created	Container created
19:58:54	openshift-ingress	kubelet	istiod-openshift-gateway-7ddd9cfdc8-z7cw2	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod
19:58:54	openshift-ingress	multus	istiod-openshift-gateway-7ddd9cfdc8-z7cw2	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
19:58:55	openshift-ingress	deployment-controller	istiod-openshift-gateway	ScalingReplicaSet	Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0
19:58:55	opendatahub	kubelet	model-serving-api-75bd89fdfc-svr6c	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 1.775s (1.775s including waiting). Image size: 184925761 bytes.
19:58:55	opendatahub	kubelet	model-serving-api-75bd89fdfc-svr6c	Created	Container created
19:58:55	openshift-ingress	replicaset-controller	istiod-openshift-gateway-7876bfcb74	SuccessfulDelete	Deleted pod: istiod-openshift-gateway-7876bfcb74-w4fzz
19:58:55	opendatahub	kubelet	model-serving-api-75bd89fdfc-svr6c	Started	Container started
19:58:55	openshift-ingress	kubelet	istiod-openshift-gateway-7876bfcb74-w4fzz	Killing	Stopping container discovery
19:58:56	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-scdxv	Created	Container created
19:58:56	opendatahub	kubelet	kserve-controller-manager-776cc66d47-hzzhx	Created	Container created
19:58:56	opendatahub	kubelet	odh-model-controller-769875db56-lm88t	Started	Container started
19:58:56	opendatahub	kubelet	odh-model-controller-769875db56-lm88t	Created	Container created
19:58:56	opendatahub	kubelet	odh-model-controller-769875db56-lm88t	Pulled	Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.146s (3.146s including waiting). Image size: 236755567 bytes.
19:58:56	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-scdxv	Started	Container started
19:58:56	opendatahub	odh-model-controller-769875db56-lm88t_2e56fc31-a9c9-4c61-aae1-50ffd60a3020	odh-model-controller.opendatahub.io	LeaderElection	odh-model-controller-769875db56-lm88t_2e56fc31-a9c9-4c61-aae1-50ffd60a3020 became leader
19:58:56	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-scdxv	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.307s (3.307s including waiting). Image size: 235100981 bytes.
19:58:56	opendatahub	kubelet	kserve-controller-manager-776cc66d47-hzzhx	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 2.855s (2.855s including waiting). Image size: 247676581 bytes.
19:58:56	opendatahub	kserve-controller-manager-776cc66d47-hzzhx_50a90235-5345-4f52-a4d2-c73a12f799fb	kserve-controller-manager-leader-lock	LeaderElection	kserve-controller-manager-776cc66d47-hzzhx_50a90235-5345-4f52-a4d2-c73a12f799fb became leader
19:58:56	opendatahub	kubelet	kserve-controller-manager-776cc66d47-hzzhx	Started	Container started
19:58:57	opendatahub	llmisvc-controller-manager-869d5dcc5d-scdxv_8161ec18-e757-4705-bfa3-d1406dfb8bed	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-869d5dcc5d-scdxv_8161ec18-e757-4705-bfa3-d1406dfb8bed became leader
19:59:13 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:13 (x4)	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:28	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:28	openshift-ingress	horizontal-pod-autoscaler	istiod-openshift-gateway	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:47	opendatahub	replicaset-controller	maas-controller-655c897cd5	SuccessfulCreate	Created pod: maas-controller-655c897cd5-pf8cd
19:59:47	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-655c897cd5 from 0 to 1
19:59:48	opendatahub	multus	maas-controller-655c897cd5-pf8cd	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
19:59:48	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled up replica set maas-controller-5655cc7547 from 0 to 1
19:59:48	opendatahub	replicaset-controller	maas-controller-5655cc7547	SuccessfulCreate	Created pod: maas-controller-5655cc7547-q2mfq
19:59:48	opendatahub	kubelet	maas-controller-655c897cd5-pf8cd	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:58f4bd62b528cf12545ff590761acd8e9477c112e987d7216e19f556835ad5df"
19:59:49	opendatahub	kubelet	maas-controller-5655cc7547-q2mfq	Pulling	Pulling image "quay.io/opendatahub/maas-controller@sha256:58f4bd62b528cf12545ff590761acd8e9477c112e987d7216e19f556835ad5df"
19:59:49	opendatahub	multus	maas-controller-5655cc7547-q2mfq	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
19:59:51	opendatahub	kubelet	maas-controller-5655cc7547-q2mfq	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:58f4bd62b528cf12545ff590761acd8e9477c112e987d7216e19f556835ad5df" in 2.561s (2.561s including waiting). Image size: 200099314 bytes.
19:59:51	opendatahub	kubelet	maas-controller-655c897cd5-pf8cd	Pulled	Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:58f4bd62b528cf12545ff590761acd8e9477c112e987d7216e19f556835ad5df" in 3.06s (3.06s including waiting). Image size: 200099314 bytes.
19:59:51	opendatahub	kubelet	maas-controller-5655cc7547-q2mfq	Created	Container created
19:59:51	opendatahub	kubelet	maas-controller-5655cc7547-q2mfq	Started	Container started
19:59:51	opendatahub	kubelet	maas-controller-655c897cd5-pf8cd	Started	Container started
19:59:51	opendatahub	kubelet	maas-controller-655c897cd5-pf8cd	Created	Container created
19:59:52	opendatahub	maas-controller-655c897cd5-pf8cd_fff8de70-c348-4ce8-99e5-5d4c5a76757e	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-655c897cd5-pf8cd_fff8de70-c348-4ce8-99e5-5d4c5a76757e became leader
19:59:53	opendatahub	replicaset-controller	usage-logs-tenancy-proxy-9c5d49bcd	SuccessfulCreate	Created pod: usage-logs-tenancy-proxy-9c5d49bcd-r9d4h
19:59:53	opendatahub	deployment-controller	usage-logs-tenancy-proxy	ScalingReplicaSet	Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1
19:59:54	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	Pulling	Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172"
19:59:54	opendatahub	multus	usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
20:00:02	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:00:02 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Created	Container created
20:00:02 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Started	Container started
20:00:03	opendatahub	deployment-controller	maas-controller	ScalingReplicaSet	Scaled down replica set maas-controller-655c897cd5 from 1 to 0
20:00:03	opendatahub	kubelet	maas-controller-655c897cd5-pf8cd	Killing	Stopping container manager
20:00:03	opendatahub	replicaset-controller	maas-controller-655c897cd5	SuccessfulDelete	Deleted pod: maas-controller-655c897cd5-pf8cd
20:00:07 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Unhealthy	Startup probe failed: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused
20:00:07 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	ProbeError	Startup probe error: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused body:
20:00:08	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	Pulled	Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 14.07s (14.07s including waiting). Image size: 1089502687 bytes.
20:00:08	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	Created	Container created
20:00:08	opendatahub	kubelet	usage-logs-tenancy-proxy-9c5d49bcd-r9d4h	Started	Container started
20:00:18 (x4)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:18 (x4)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:19	opendatahub	maas-controller-5655cc7547-q2mfq_37f9004c-5ef6-495c-9143-d6050a02af6e	maas-controller.models-as-a-service.opendatahub.io	LeaderElection	maas-controller-5655cc7547-q2mfq_37f9004c-5ef6-495c-9143-d6050a02af6e became leader
20:00:33	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6f6944bcdd from 0 to 1
20:00:33 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:33 (x2)	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:34	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-6889cbd8c4 from 0 to 1
20:00:34	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulCreate	Created pod: authorino-6889cbd8c4-pbgz6
20:00:34	kuadrant-system	multus	authorino-94899989-mcjbn	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
20:00:34	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6f6944bcdd from 1 to 0
20:00:34	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulDelete	Deleted pod: authorino-6f6944bcdd-zhjjl
20:00:34	kuadrant-system	kubelet	authorino-6889cbd8c4-pbgz6	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:00:34	kuadrant-system	replicaset-controller	authorino-6889cbd8c4	SuccessfulDelete	Deleted pod: authorino-6889cbd8c4-pbgz6
20:00:34	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled up replica set authorino-94899989 from 0 to 1
20:00:34	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-6889cbd8c4 from 1 to 0
20:00:34	kuadrant-system	multus	authorino-6889cbd8c4-pbgz6	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
20:00:34	kuadrant-system	replicaset-controller	authorino-6f6944bcdd	SuccessfulCreate	Created pod: authorino-6f6944bcdd-zhjjl
20:00:34	kuadrant-system	replicaset-controller	authorino-94899989	SuccessfulCreate	Created pod: authorino-94899989-mcjbn
20:00:35	kuadrant-system	kubelet	authorino-94899989-mcjbn	Created	Container created
20:00:35	kuadrant-system	kubelet	authorino-94899989-mcjbn	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:00:35	kuadrant-system	kubelet	authorino-94899989-mcjbn	Started	Container started
20:00:35	kuadrant-system	kubelet	authorino-94899989-mcjbn	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 744ms (744ms including waiting). Image size: 204096810 bytes.
20:00:36	kuadrant-system	kubelet	authorino-56c6c9ddd5-fngsn	Killing	Stopping container authorino
20:00:36	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	Scaled down replica set authorino-56c6c9ddd5 from 1 to 0
20:00:36	kuadrant-system	replicaset-controller	authorino-56c6c9ddd5	SuccessfulDelete	Deleted pod: authorino-56c6c9ddd5-fngsn
20:00:48	kuadrant-system	kubelet	authorino-6889cbd8c4-pbgz6	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 14.14s (14.14s including waiting). Image size: 204096810 bytes.
20:00:48	kuadrant-system	kubelet	authorino-6889cbd8c4-pbgz6	Started	Container started
20:00:48	kuadrant-system	kubelet	authorino-6889cbd8c4-pbgz6	Created	Container created
20:00:49	kuadrant-system	kubelet	authorino-6889cbd8c4-pbgz6	Killing	Stopping container authorino
20:01:19	kuadrant-system	replicaset-controller	authorino-555b588c49	SuccessfulCreate	Created pod: authorino-555b588c49-rg2lv
20:01:19	kuadrant-system	multus	authorino-555b588c49-rg2lv	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
20:01:19	kuadrant-system	kubelet	authorino-555b588c49-rg2lv	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:01:20	kuadrant-system	kubelet	authorino-555b588c49-rg2lv	Started	Container started
20:01:20	kuadrant-system	kubelet	authorino-555b588c49-rg2lv	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 999ms (999ms including waiting). Image size: 204096810 bytes.
20:01:20	kuadrant-system	kubelet	authorino-555b588c49-rg2lv	Created	Container created
20:01:21	kuadrant-system	replicaset-controller	authorino-94899989	SuccessfulDelete	Deleted pod: authorino-94899989-mcjbn
20:01:21	opendatahub	replicaset-controller	llmisvc-controller-manager-6466c554d8	SuccessfulCreate	Created pod: llmisvc-controller-manager-6466c554d8-d6fbl
20:01:21	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1
20:01:21	kuadrant-system	kubelet	authorino-94899989-mcjbn	Killing	Stopping container authorino
20:01:22	opendatahub	multus	llmisvc-controller-manager-6466c554d8-d6fbl	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
20:01:22	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-d6fbl	Pulling	Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"
20:01:22	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-d6fbl	Started	Container started
20:01:22	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-d6fbl	Created	Container created
20:01:22	opendatahub	kubelet	llmisvc-controller-manager-6466c554d8-d6fbl	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 407ms (407ms including waiting). Image size: 235100981 bytes.
20:01:54	opendatahub	kubelet	llmisvc-controller-manager-869d5dcc5d-scdxv	Killing	Stopping container manager
20:01:54	opendatahub	deployment-controller	llmisvc-controller-manager	ScalingReplicaSet	Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0
20:01:54	opendatahub	replicaset-controller	llmisvc-controller-manager-869d5dcc5d	SuccessfulDelete	Deleted pod: llmisvc-controller-manager-869d5dcc5d-scdxv
20:02:08	opendatahub	llmisvc-controller-manager-6466c554d8-d6fbl_cd0b1389-c1c8-4df2-b064-ac06c85d1255	llminferenceservice-kserve-controller-manager	LeaderElection	llmisvc-controller-manager-6466c554d8-d6fbl_cd0b1389-c1c8-4df2-b064-ac06c85d1255 became leader
20:02:10	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-5d8dddb676 from 0 to 1
20:02:10	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulCreate	Created pod: ai-gateway-controller-5d8dddb676-w4rzz
20:02:11	opendatahub	multus	ai-gateway-controller-5d8dddb676-w4rzz	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
20:02:11	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-w4rzz	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
20:02:13	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-w4rzz	Started	Container started
20:02:13	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-w4rzz	Created	Container created
20:02:13	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-w4rzz	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 2.038s (2.038s including waiting). Image size: 184136103 bytes.
20:02:14	opendatahub	ai-gateway-controller-5d8dddb676-w4rzz_d32b717a-9e76-4d48-ba75-f7fdbd01fe30	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-5d8dddb676-w4rzz_d32b717a-9e76-4d48-ba75-f7fdbd01fe30 became leader
20:02:15	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	SuccessfulCreate	Created pod: payload-pre-processing-7b98bb678c-zl76j
20:02:15	openshift-ingress	multus	payload-pre-processing-7b98bb678c-zl76j	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:02:15	openshift-ingress	deployment-controller	payload-processing	ScalingReplicaSet	Scaled up replica set payload-processing-7bb9489676 from 0 to 1
20:02:15	openshift-ingress	multus	payload-processing-7bb9489676-7mbkz	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
20:02:15	openshift-ingress	kubelet	payload-processing-7bb9489676-7mbkz	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:02:15	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-zl76j	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:02:15	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	SuccessfulCreate	Created pod: payload-processing-7bb9489676-7mbkz
20:02:15	openshift-ingress	deployment-controller	payload-pre-processing	ScalingReplicaSet	Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
20:02:15 (x5)	openshift-ingress	replicaset-controller	payload-processing-7bb9489676	FailedCreate	Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:02:15 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-7b98bb678c	FailedCreate	Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
20:02:18	openshift-ingress	kubelet	payload-processing-7bb9489676-7mbkz	Started	Container started
20:02:18	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-zl76j	Started	Container started
20:02:18	openshift-ingress	kubelet	payload-processing-7bb9489676-7mbkz	Created	Container created
20:02:18	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-zl76j	Created	Container created
20:02:18	openshift-ingress	kubelet	payload-pre-processing-7b98bb678c-zl76j	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.328s (2.328s including waiting). Image size: 125353963 bytes.
20:02:18	openshift-ingress	kubelet	payload-processing-7bb9489676-7mbkz	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.381s (2.381s including waiting). Image size: 125353963 bytes.
20:02:25	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled up replica set ai-gateway-controller-76f789867f from 0 to 1
20:02:25	opendatahub	replicaset-controller	ai-gateway-controller-76f789867f	SuccessfulCreate	Created pod: ai-gateway-controller-76f789867f-ftf5k
20:02:26	opendatahub	kubelet	ai-gateway-controller-76f789867f-ftf5k	Created	Container created
20:02:26	opendatahub	kubelet	ai-gateway-controller-76f789867f-ftf5k	Started	Container started
20:02:26	opendatahub	kubelet	ai-gateway-controller-76f789867f-ftf5k	Pulled	Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 422ms (422ms including waiting). Image size: 184136103 bytes.
20:02:26	opendatahub	kubelet	ai-gateway-controller-76f789867f-ftf5k	Pulling	Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"
20:02:26	opendatahub	multus	ai-gateway-controller-76f789867f-ftf5k	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
20:02:38	opendatahub	replicaset-controller	ai-gateway-controller-5d8dddb676	SuccessfulDelete	Deleted pod: ai-gateway-controller-5d8dddb676-w4rzz
20:02:38	opendatahub	kubelet	ai-gateway-controller-5d8dddb676-w4rzz	Killing	Stopping container manager
20:02:38	opendatahub	deployment-controller	ai-gateway-controller	ScalingReplicaSet	Scaled down replica set ai-gateway-controller-5d8dddb676 from 1 to 0
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine-default
20:02:45	llm	replicaset-controller	e2e-distinct-2-simulated-kserve-cbd4f5586	SuccessfulCreate	Created pod: e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-2-simulated-kserve-workload-svc
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-2-simulated-kserve-route
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Service llm/e2e-distinct-2-simulated-kserve-workload-svc
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Deployment llm/e2e-distinct-2-simulated-kserve
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-2-simulated-prometheus-scraping
20:02:45	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Created	Created v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
20:02:45	llm	deployment-controller	e2e-distinct-2-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-2-simulated-kserve-cbd4f5586 from 0 to 1
20:02:46	llm	multus	e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
20:02:46	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:47	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.422s (1.422s including waiting). Image size: 98346788 bytes.
20:02:47	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	Created	Container created
20:02:47	llm	kubelet	e2e-distinct-2-simulated-kserve-cbd4f5586-c96ks	Started	Container started
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine-default
20:02:48	llm	multus	e2e-distinct-simulated-kserve-7b998b79d9-5qx79	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
20:02:48	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-5qx79	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.HTTPRoute llm/e2e-distinct-simulated-kserve-route
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Service llm/e2e-distinct-simulated-kserve-workload-svc
20:02:48	llm	replicaset-controller	e2e-distinct-simulated-kserve-7b998b79d9	SuccessfulCreate	Created pod: e2e-distinct-simulated-kserve-7b998b79d9-5qx79
20:02:48	llm	deployment-controller	e2e-distinct-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-distinct-simulated-kserve-7b998b79d9 from 0 to 1
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.NetworkPolicy llm/e2e-distinct-simulated-prometheus-scraping
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.Deployment llm/e2e-distinct-simulated-kserve
20:02:48	llm	LLMInferenceServiceController	e2e-distinct-simulated	Created	Created v1.DestinationRule llm/e2e-distinct-simulated-kserve-workload-svc
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
20:02:50	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-5qx79	Started	Container started
20:02:50	llm	deployment-controller	e2e-embedding-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-embedding-simulated-kserve-67bf594cd6 from 0 to 1
20:02:50	llm	replicaset-controller	e2e-embedding-simulated-kserve-67bf594cd6	SuccessfulCreate	Created pod: e2e-embedding-simulated-kserve-67bf594cd6-cjsx6
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Deployment llm/e2e-embedding-simulated-kserve
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.Service llm/e2e-embedding-simulated-kserve-workload-svc
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.HTTPRoute llm/e2e-embedding-simulated-kserve-route
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.DestinationRule llm/e2e-embedding-simulated-kserve-workload-svc
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine-default
20:02:50	llm	multus	e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
20:02:50	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-5qx79	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.525s (1.525s including waiting). Image size: 98346788 bytes.
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine
20:02:50	llm	kubelet	e2e-distinct-simulated-kserve-7b998b79d9-5qx79	Created	Container created
20:02:50	llm	LLMInferenceServiceController	e2e-embedding-simulated	Created	Created v1.NetworkPolicy llm/e2e-embedding-simulated-prometheus-scraping
20:02:50	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:51	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 337ms (337ms including waiting). Image size: 98346788 bytes.
20:02:51	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	Created	Container created
20:02:51	llm	kubelet	e2e-embedding-simulated-kserve-67bf594cd6-cjsx6	Started	Container started
20:02:52	llm	deployment-controller	e2e-trlp-test-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-trlp-test-simulated-kserve-69b588897d from 0 to 1
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.HTTPRoute llm/e2e-trlp-test-simulated-kserve-route
20:02:52	llm	replicaset-controller	e2e-trlp-test-simulated-kserve-69b588897d	SuccessfulCreate	Created pod: e2e-trlp-test-simulated-kserve-69b588897d-mg8s7
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.NetworkPolicy llm/e2e-trlp-test-simulated-prometheus-scraping
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine-default
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.DestinationRule llm/e2e-trlp-test-simulated-kserve-workload-svc
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Deployment llm/e2e-trlp-test-simulated-kserve
20:02:52	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Created	Created v1.Service llm/e2e-trlp-test-simulated-kserve-workload-svc
20:02:53	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	Created	Container created
20:02:53	llm	multus	e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
20:02:53	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:53	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 334ms (334ms including waiting). Image size: 98346788 bytes.
20:02:53	opendatahub	ai-gateway-controller-76f789867f-ftf5k_e2f4215b-3e8e-4037-abf8-4c573ef25d90	ai-gateway-controller-leader.opendatahub.io	LeaderElection	ai-gateway-controller-76f789867f-ftf5k_e2f4215b-3e8e-4037-abf8-4c573ef25d90 became leader
20:02:53	llm	kubelet	e2e-trlp-test-simulated-kserve-69b588897d-mg8s7	Started	Container started
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-route
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Service llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
20:02:55	llm	replicaset-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc	SuccessfulCreate	Created pod: e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw
20:02:55	llm	multus	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
20:02:55	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:02:55	llm	deployment-controller	e2e-unconfigured-facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc from 0 to 1
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-prometheus-scraping
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/e2e-unconab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/eab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine-default
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc
20:02:55	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Created	Created v1.Deployment llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve
20:02:56	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 349ms (349ms including waiting). Image size: 98346788 bytes.
20:02:56	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	Started	Container started
20:02:56	llm	kubelet	e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m8sfw	Created	Container created
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.DestinationRule llm/echo-simulator-kserve-workload-svc
20:02:57	llm	replicaset-controller	echo-simulator-kserve-64b6f8f96c	SuccessfulCreate	Created pod: echo-simulator-kserve-64b6f8f96c-kgwfb
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.NetworkPolicy llm/echo-simulator-prometheus-scraping
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Secret llm/echo-simulator-kserve-self-signed-certs
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Deployment llm/echo-simulator-kserve
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine-default
20:02:57	llm	deployment-controller	echo-simulator-kserve	ScalingReplicaSet	Scaled up replica set echo-simulator-kserve-64b6f8f96c from 0 to 1
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.Service llm/echo-simulator-kserve-workload-svc
20:02:57	llm	LLMInferenceServiceController	echo-simulator	Created	Created v1.HTTPRoute llm/echo-simulator-kserve-route
20:02:58	llm	multus	echo-simulator-kserve-64b6f8f96c-kgwfb	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
20:02:58	llm	kubelet	echo-simulator-kserve-64b6f8f96c-kgwfb	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0"
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.DestinationRule llm/facebook-opt-125m-simulated-kserve-workload-svc
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.HTTPRoute llm/facebook-opt-125m-simulated-kserve-route
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Service llm/facebook-opt-125m-simulated-kserve-workload-svc
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.NetworkPolicy llm/facebook-opt-125m-simulated-prometheus-scraping
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.Deployment llm/facebook-opt-125m-simulated-kserve
20:02:59	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Created	Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine-default
20:02:59	llm	multus	facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
20:02:59	llm	deployment-controller	facebook-opt-125m-simulated-kserve	ScalingReplicaSet	Scaled up replica set facebook-opt-125m-simulated-kserve-84b4ff656c from 0 to 1
20:02:59	llm	replicaset-controller	facebook-opt-125m-simulated-kserve-84b4ff656c	SuccessfulCreate	Created pod: facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5
20:02:59	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
20:03:00	llm	replicaset-controller	premium-simulated-simulated-premium-kserve-5f5c4b459f	SuccessfulCreate	Created pod: premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm
20:03:00	llm	kubelet	echo-simulator-kserve-64b6f8f96c-kgwfb	Started	Container started
20:03:00	llm	kubelet	echo-simulator-kserve-64b6f8f96c-kgwfb	Created	Container created
20:03:00	llm	kubelet	echo-simulator-kserve-64b6f8f96c-kgwfb	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0" in 1.762s (1.762s including waiting). Image size: 78208878 bytes.
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.NetworkPolicy llm/premium-simulated-simulated-premium-prometheus-scraping
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/premium-simulated-simulated-premium-kserve-llmisvc-engine
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.PodMonitor llm/p75d89895ad19b4ac62612d08bfe7f574-kserve-llmisvc-engine-default
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.DestinationRule llm/premium-simulated-simulated-premium-kserve-workload-svc
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.HTTPRoute llm/premium-simulated-simulated-premium-kserve-route
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Service llm/premium-simulated-simulated-premium-kserve-workload-svc
20:03:00	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Created	Created v1.Deployment llm/premium-simulated-simulated-premium-kserve
20:03:00	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 356ms (356ms including waiting). Image size: 98346788 bytes.
20:03:00	llm	deployment-controller	premium-simulated-simulated-premium-kserve	ScalingReplicaSet	Scaled up replica set premium-simulated-simulated-premium-kserve-5f5c4b459f from 0 to 1
20:03:00	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	Started	Container started
20:03:00	llm	kubelet	facebook-opt-125m-simulated-kserve-84b4ff656c-dncg5	Created	Container created
20:03:01	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
20:03:01 (x2)	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	Updated	Updated v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs
20:03:01	llm	multus	premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
20:03:01	llm	LLMInferenceServiceController	e2e-distinct-2-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-2-simulated] is Ready
20:03:01	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 336ms (336ms including waiting). Image size: 98346788 bytes.
20:03:01	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	Created	Container created
20:03:01	llm	kubelet	premium-simulated-simulated-premium-kserve-5f5c4b459f-g72mm	Started	Container started
20:03:02	llm	LLMInferenceServiceController	e2e-distinct-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-distinct-simulated] is Ready
20:03:02	llm	LLMInferenceServiceController	e2e-distinct-simulated	Updated	Updated v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs
20:03:03	llm	LLMInferenceServiceController	e2e-embedding-simulated	Updated	Updated v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs
20:03:03	llm	LLMInferenceServiceController	e2e-embedding-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-embedding-simulated] is Ready
20:03:05	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	Updated	Updated v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs
20:03:05	llm	LLMInferenceServiceController	e2e-trlp-test-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-trlp-test-simulated] is Ready
20:03:06	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs
20:03:06	llm	LLMInferenceServiceController	e2e-unconfigured-facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [e2e-unconfigured-facebook-opt-125m-simulated] is Ready
20:03:08	llm	LLMInferenceServiceController	echo-simulator	Updated	Updated v1.Secret llm/echo-simulator-kserve-self-signed-certs
20:03:09	llm	LLMInferenceServiceController	echo-simulator	LLMInferenceServiceReady	LLMInferenceService [echo-simulator] is Ready
20:03:09	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	Updated	Updated v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs
20:03:10	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	Updated	Updated v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs
20:03:11	llm	LLMInferenceServiceController	facebook-opt-125m-simulated	LLMInferenceServiceReady	LLMInferenceService [facebook-opt-125m-simulated] is Ready
20:03:12	llm	LLMInferenceServiceController	premium-simulated-simulated-premium	LLMInferenceServiceReady	LLMInferenceService [premium-simulated-simulated-premium] is Ready
20:03:13	kuadrant-system	replicaset-controller	authorino-66ccd8f8b	SuccessfulCreate	Created pod: authorino-66ccd8f8b-vgm8p
20:03:14	kuadrant-system	multus	authorino-66ccd8f8b-vgm8p	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
20:03:14	kuadrant-system	kubelet	authorino-66ccd8f8b-vgm8p	Pulling	Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"
20:03:14	kuadrant-system	kubelet	authorino-66ccd8f8b-vgm8p	Pulled	Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 886ms (886ms including waiting). Image size: 204096810 bytes.
20:03:15	kuadrant-system	kubelet	authorino-66ccd8f8b-vgm8p	Started	Container started
20:03:15	kuadrant-system	kubelet	authorino-66ccd8f8b-vgm8p	Created	Container created
20:03:15	kuadrant-system	replicaset-controller	authorino-555b588c49	SuccessfulDelete	Deleted pod: authorino-555b588c49-rg2lv
20:03:15	kuadrant-system	kubelet	authorino-555b588c49-rg2lv	Killing	Stopping container authorino
20:03:15 (x4)	kuadrant-system	deployment-controller	authorino	ScalingReplicaSet	(combined from similar events): Scaled down replica set authorino-555b588c49 from 1 to 0
20:03:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
20:03:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-8vwd6	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
20:03:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-8vwd6
20:03:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-vft4f
20:03:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-5xhnp	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
20:03:18	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 1 to 4
20:03:18	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 4; reason: cpu resource utilization (percentage of request) above target
20:03:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-vft4f	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:03:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-5xhnp
20:03:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
20:03:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Pulling	Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
20:03:22	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Created	Container created
20:03:22	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.679s (3.679s including waiting). Image size: 183555333 bytes.
20:03:22	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Started	Container started
20:03:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.661s (4.661s including waiting). Image size: 183555333 bytes.
20:03:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Started	Container started
20:03:23	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Created	Container created
20:03:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Started	Container started
20:03:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Created	Container created
20:03:25	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Pulled	Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 6.303s (6.303s including waiting). Image size: 183555333 bytes.
20:03:29 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Unhealthy	Startup probe failed: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused
20:03:29 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	ProbeError	Startup probe error: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused body:
20:03:30 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Unhealthy	Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused
20:03:30 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	ProbeError	Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused body:
20:03:32 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Unhealthy	Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused
20:03:32 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	ProbeError	Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused body:
20:03:33	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-9hc8b	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
20:03:33	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 7; reason: cpu resource utilization (percentage of request) above target
20:03:33	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-9hc8b
20:03:33	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-wrztq
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Started	Container started
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Created	Container created
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:03:33	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-kxp6g	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
20:03:33	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-wrztq	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:03:33	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 4 to 7
20:03:33	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-kxp6g
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Created	Container created
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Started	Container started
20:03:33	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:03:34	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Created	Container created
20:03:34	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Started	Container started
20:03:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	ProbeError	Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused body:
20:03:40 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Unhealthy	Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused
20:03:40 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	ProbeError	Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused body:
20:03:40 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Unhealthy	Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused
20:03:41 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	ProbeError	Startup probe error: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused body:
20:03:41 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Unhealthy	Startup probe failed: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.PodMonitor llm/grid-inv-ba617b3a-model-kserve-llmisvc-engine-default
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.HTTPRoute llm/grid-inv-ba617b3a-model-kserve-route
20:03:53	llm	multus	grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
20:03:53	llm	kubelet	grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	Created	Container created
20:03:53	llm	kubelet	grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	Started	Container started
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.Service llm/grid-inv-ba617b3a-model-kserve-workload-svc
20:03:53	llm	kubelet	grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:03:53	llm	replicaset-controller	grid-inv-ba617b3a-model-kserve-759ccc5c6c	SuccessfulCreate	Created pod: grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.NetworkPolicy llm/grid-inv-ba617b3a-model-prometheus-scraping
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.Deployment llm/grid-inv-ba617b3a-model-kserve
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.DestinationRule llm/grid-inv-ba617b3a-model-kserve-workload-svc
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.PodMonitor llm/grid-inv-ba617b3a-model-kserve-llmisvc-engine
20:03:53	llm	deployment-controller	grid-inv-ba617b3a-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-ba617b3a-model-kserve-759ccc5c6c from 0 to 1
20:03:53	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Created	Created v1.Secret llm/grid-inv-ba617b3a-model-kserve-self-signed-certs
20:03:55	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	Updated	Updated v1.Secret llm/grid-inv-ba617b3a-model-kserve-self-signed-certs
20:03:56	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.Secret llm/grid-inv-ba617b3a-unmatched-kserve-self-signed-certs
20:03:57	llm	kubelet	grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	Created	Container created
20:03:57	llm	kubelet	grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	Started	Container started
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.PodMonitor llm/grid-inv-ba617b3a-unmatched-kserve-llmisvc-engine-default
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-ba617b3a-unmatched-prometheus-scraping
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.DestinationRule llm/grid-inv-ba617b3a-unmatched-kserve-workload-svc
20:03:57	llm	kubelet	grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.Service llm/grid-inv-ba617b3a-unmatched-kserve-workload-svc
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.Deployment llm/grid-inv-ba617b3a-unmatched-kserve
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-ba617b3a-unmatched-kserve-route
20:03:57	llm	multus	grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
20:03:57	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Created	Created v1.PodMonitor llm/grid-inv-ba617b3a-unmatched-kserve-llmisvc-engine
20:03:57	llm	replicaset-controller	grid-inv-ba617b3a-unmatched-kserve-8546586745	SuccessfulCreate	Created pod: grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk
20:03:57	llm	deployment-controller	grid-inv-ba617b3a-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-ba617b3a-unmatched-kserve-8546586745 from 0 to 1
20:04:01 (x2)	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	Updated	Updated v1.Secret llm/grid-inv-ba617b3a-unmatched-kserve-self-signed-certs
20:04:04	llm	LLMInferenceServiceController	grid-inv-ba617b3a-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-ba617b3a-model] is Ready
20:04:08	llm	LLMInferenceServiceController	grid-inv-ba617b3a-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-ba617b3a-unmatched] is Ready
20:04:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:18	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 10; reason: cpu resource utilization (percentage of request) above target
20:04:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-f6z8p	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
20:04:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-f6z8p
20:04:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	Created pod: maas-default-gateway-openshift-default-67bf986f99-wgkhr
20:04:18	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulCreate	(combined from similar events): Created pod: maas-default-gateway-openshift-default-67bf986f99-88lvc
20:04:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-wgkhr	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
20:04:18	openshift-ingress	multus	maas-default-gateway-openshift-default-67bf986f99-88lvc	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
20:04:18	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:04:18	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 7 to 10
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Created	Container created
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Started	Container started
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Started	Container started
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Created	Container created
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Started	Container started
20:04:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Created	Container created
20:04:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	ProbeError	Startup probe error: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused body:
20:04:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	ProbeError	Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused body:
20:04:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Unhealthy	Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused
20:04:26 (x7)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Unhealthy	Startup probe failed: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused
20:04:27 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Unhealthy	Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused
20:04:27 (x8)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	ProbeError	Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused body:
20:04:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Unhealthy	Startup probe failed: HTTP probe failed with statuscode: 503
20:04:30	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	ProbeError	Startup probe error: HTTP probe failed with statuscode: 503 body:
20:04:32 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Unhealthy	Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:04:32 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Unhealthy	Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:04:32 (x5)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Unhealthy	Startup probe failed: Get "http://10.133.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:04:33 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	ProbeError	Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:04:33 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	ProbeError	Startup probe error: Get "http://10.133.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:04:33 (x6)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	ProbeError	Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:05:05	llm	kubelet	grid-inv-ba617b3a-model-kserve-759ccc5c6c-mwq25	Killing	Stopping container main
20:05:06	llm	kubelet	grid-inv-ba617b3a-unmatched-kserve-8546586745-r8rqk	Killing	Stopping container main
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.HTTPRoute llm/grid-inv-14f915b6-model-kserve-route
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.Deployment llm/grid-inv-14f915b6-model-kserve
20:05:09	llm	replicaset-controller	grid-inv-14f915b6-model-kserve-8c9755b54	SuccessfulCreate	Created pod: grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7
20:05:09	llm	deployment-controller	grid-inv-14f915b6-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-14f915b6-model-kserve-8c9755b54 from 0 to 1
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.PodMonitor llm/grid-inv-14f915b6-model-kserve-llmisvc-engine
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.PodMonitor llm/grid-inv-14f915b6-model-kserve-llmisvc-engine-default
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.DestinationRule llm/grid-inv-14f915b6-model-kserve-workload-svc
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.Service llm/grid-inv-14f915b6-model-kserve-workload-svc
20:05:09	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.Secret llm/grid-inv-14f915b6-model-kserve-self-signed-certs
20:05:10	llm	kubelet	grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	Started	Container started
20:05:10	llm	kubelet	grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:05:10	llm	kubelet	grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	Created	Container created
20:05:10	llm	multus	grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
20:05:10	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Created	Created v1.NetworkPolicy llm/grid-inv-14f915b6-model-prometheus-scraping
20:05:12	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.Secret llm/grid-inv-14f915b6-unmatched-kserve-self-signed-certs
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.PodMonitor llm/grid-inv-14f915b6-unmatched-kserve-llmisvc-engine
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.DestinationRule llm/grid-inv-14f915b6-unmatched-kserve-workload-svc
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.PodMonitor llm/grid-inv-14f915b6-unmatched-kserve-llmisvc-engine-default
20:05:13	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Pulling	Pulling image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1"
20:05:13	llm	multus	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-14f915b6-unmatched-prometheus-scraping
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-14f915b6-unmatched-kserve-route
20:05:13	llm	replicaset-controller	grid-inv-14f915b6-unmatched-kserve-d4545864	SuccessfulCreate	Created pod: grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf
20:05:13	llm	deployment-controller	grid-inv-14f915b6-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-14f915b6-unmatched-kserve-d4545864 from 0 to 1
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.Service llm/grid-inv-14f915b6-unmatched-kserve-workload-svc
20:05:13	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Created	Created v1.Deployment llm/grid-inv-14f915b6-unmatched-kserve
20:05:15	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	Updated	Updated v1.Secret llm/grid-inv-14f915b6-model-kserve-self-signed-certs
20:05:16	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Created	Container created
20:05:16	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Started	Container started
20:05:16	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Pulled	Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" in 2.684s (2.684s including waiting). Image size: 98346788 bytes.
20:05:18	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	Updated	Updated v1.Secret llm/grid-inv-14f915b6-unmatched-kserve-self-signed-certs
20:05:20	llm	LLMInferenceServiceController	grid-inv-14f915b6-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-14f915b6-model] is Ready
20:05:24	llm	LLMInferenceServiceController	grid-inv-14f915b6-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-14f915b6-unmatched] is Ready
20:05:27	opendatahub	kubelet	test-curl-8726-5c62b6	Pulling	Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"
20:05:27	opendatahub	multus	test-curl-8726-5c62b6	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
20:05:30	opendatahub	kubelet	test-curl-8726-5c62b6	Pulled	Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 2.576s (2.576s including waiting). Image size: 106100986 bytes.
20:05:30	opendatahub	kubelet	test-curl-8726-5c62b6	Created	Container created
20:05:30	opendatahub	kubelet	test-curl-8726-5c62b6	Started	Container started
20:05:32	opendatahub	kubelet	test-curl-8726-77e414	Started	Container started
20:05:32	opendatahub	kubelet	test-curl-8726-77e414	Created	Container created
20:05:32	opendatahub	kubelet	test-curl-8726-77e414	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
20:05:32	opendatahub	multus	test-curl-8726-77e414	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
20:05:32	opendatahub	kubelet	test-curl-8726-5c62b6	Killing	Stopping container test-curl-8726-5c62b6
20:05:34	opendatahub	kubelet	test-curl-8726-952975	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
20:05:34	opendatahub	kubelet	test-curl-8726-952975	Started	Container started
20:05:34	opendatahub	kubelet	test-curl-8726-952975	Created	Container created
20:05:34	opendatahub	kubelet	test-curl-8726-77e414	Killing	Stopping container test-curl-8726-77e414
20:05:34	opendatahub	multus	test-curl-8726-952975	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
20:05:36	openshift-ingress	deployment-controller	e2e-shared-a-main-18ecf5-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-a-main-18ecf5-openshift-default-5c586494c from 0 to 1
20:05:36	opendatahub	kubelet	test-curl-8726-952975	Killing	Stopping container test-curl-8726-952975
20:05:36	openshift-ingress	replicaset-controller	e2e-shared-a-main-18ecf5-openshift-default-5c586494c	SuccessfulCreate	Created pod: e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s
20:05:36	openshift-ingress	gateway_labeler_controller	e2e-shared-a-main-18ecf5	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-main-18ecf5
20:05:37	openshift-ingress	multus	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
20:05:41 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-18ecf5-699646fb48	FailedCreate	Error creating: pods "payload-processing-e2e-shared-a-main-18ecf5-699646fb48-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-18ecf5: serviceaccount "payload-processing-e2e-shared-a-main-18ecf5" not found
20:05:41	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-a-main-18ecf5-699646fb48	SuccessfulCreate	Created pod: payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl
20:05:41 (x4)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf459	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf459-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-main-18ecf5: serviceaccount "payload-processing-e2e-shared-a-main-18ecf5" not found
20:05:41	openshift-ingress	multus	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
20:05:41	openshift-ingress	deployment-controller	payload-processing-e2e-shared-a-main-18ecf5	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-a-main-18ecf5-699646fb48 from 0 to 1
20:05:41	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-a-main-18ecf5	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf459 from 0 to 1
20:05:41	openshift-ingress	multus	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
20:05:41	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf459	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g
20:05:41	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Pulling	Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"
20:05:42	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Started	Container started
20:05:42	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Created	Container created
20:05:42	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Started	Container started
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Pulled	Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.325s (2.325s including waiting). Image size: 125353963 bytes.
20:05:44	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Created	Container created
20:05:50	openshift-ingress	replicaset-controller	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc	SuccessfulCreate	Created pod: e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh
20:05:50	openshift-ingress	deployment-controller	e2e-shared-b-main-dbce1a-openshift-default	ScalingReplicaSet	Scaled up replica set e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc from 0 to 1
20:05:50	openshift-ingress	gateway_labeler_controller	e2e-shared-b-main-dbce1a	AddedLabel	Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-main-dbce1a
20:05:51	openshift-ingress	multus	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
20:05:54	openshift-ingress	deployment-controller	payload-processing-e2e-shared-b-main-dbce1a	ScalingReplicaSet	Scaled up replica set payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb from 0 to 1
20:05:54 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Created	Container created
20:05:54 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:05:54	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Unhealthy	Readiness probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused
20:05:54	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	ProbeError	Readiness probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused body:
20:05:54 (x2)	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Started	Container started
20:05:55	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Created	Container created
20:05:55	openshift-ingress	deployment-controller	payload-pre-processing-e2e-shared-b-main-dbce1a	ScalingReplicaSet	Scaled up replica set payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967 from 0 to 1
20:05:55	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
20:05:55	openshift-ingress	multus	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
20:05:55	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967	SuccessfulCreate	Created pod: payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n
20:05:55	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Pulled	Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
20:05:55	openshift-ingress	multus	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
20:05:55 (x5)	openshift-ingress	replicaset-controller	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967	FailedCreate	Error creating: pods "payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-dbce1a: serviceaccount "payload-processing-e2e-shared-b-main-dbce1a" not found
20:05:55	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Started	Container started
20:05:55 (x5)	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb	FailedCreate	Error creating: pods "payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-main-dbce1a: serviceaccount "payload-processing-e2e-shared-b-main-dbce1a" not found
20:05:55	openshift-ingress	replicaset-controller	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb	SuccessfulCreate	Created pod: payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc
20:05:56	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Started	Container started
20:05:56	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Created	Container created
20:06:00 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	ProbeError	Startup probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused body:
20:06:00 (x5)	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Unhealthy	Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused
20:06:08	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	ProbeError	Readiness probe error: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:06:08	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Unhealthy	Readiness probe failed: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:06:10 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Started	Container started
20:06:10 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Created	Container created
20:06:10 (x2)	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Pulled	Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
20:06:16 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	ProbeError	Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused body:
20:06:16 (x5)	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Unhealthy	Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused
20:06:35 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-dbce1a-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:35 (x3)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-b-main-dbce1a-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:36 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-18ecf5-openshift-default	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:36 (x4)	openshift-ingress	horizontal-pod-autoscaler	e2e-shared-a-main-18ecf5-openshift-default	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:43	opendatahub	multus	test-curl-8726-4fa8a8	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
20:06:43	opendatahub	kubelet	test-curl-8726-4fa8a8	Pulled	Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod
20:06:43	opendatahub	kubelet	test-curl-8726-4fa8a8	Created	Container created
20:06:43	opendatahub	kubelet	test-curl-8726-4fa8a8	Started	Container started
20:06:45	opendatahub	kubelet	test-curl-8726-4fa8a8	Killing	Stopping container test-curl-8726-4fa8a8
20:07:32 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	ProbeError	Readiness probe error: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:07:32 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Unhealthy	Readiness probe failed: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:07:43 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Unhealthy	Readiness probe failed: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:07:43	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	ProbeError	Readiness probe error: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:07:43 (x2)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	ProbeError	Readiness probe error: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:07:43	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Unhealthy	Readiness probe failed: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:07:51	llm	kubelet	grid-inv-14f915b6-model-kserve-8c9755b54-tmdl7	Killing	Stopping container main
20:07:53	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Killing	Stopping container main
20:07:58 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	ProbeError	Readiness probe error: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:07:58	llm	kubelet	grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	Started	Container started
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.Deployment llm/grid-inv-e58e9806-model-kserve
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.Secret llm/grid-inv-e58e9806-model-kserve-self-signed-certs
20:07:58	llm	deployment-controller	grid-inv-e58e9806-model-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-e58e9806-model-kserve-7955fd6f86 from 0 to 1
20:07:58	llm	replicaset-controller	grid-inv-e58e9806-model-kserve-7955fd6f86	SuccessfulCreate	Created pod: grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.HTTPRoute llm/grid-inv-e58e9806-model-kserve-route
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.NetworkPolicy llm/grid-inv-e58e9806-model-prometheus-scraping
20:07:58 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:07:58	llm	kubelet	grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	Created	Container created
20:07:58	llm	kubelet	grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:07:58	llm	multus	grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.DestinationRule llm/grid-inv-e58e9806-model-kserve-workload-svc
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.PodMonitor llm/grid-inv-e58e9806-model-kserve-llmisvc-engine-default
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.Service llm/grid-inv-e58e9806-model-kserve-workload-svc
20:07:58	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Created	Created v1.PodMonitor llm/grid-inv-e58e9806-model-kserve-llmisvc-engine
20:08:00	llm	deployment-controller	grid-inv-e58e9806-unmatched-kserve	ScalingReplicaSet	Scaled up replica set grid-inv-e58e9806-unmatched-kserve-5f49c885d8 from 0 to 1
20:08:00	llm	replicaset-controller	grid-inv-e58e9806-unmatched-kserve-5f49c885d8	SuccessfulCreate	Created pod: grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.PodMonitor llm/grid-inv-e58e9806-unmatched-kserve-llmisvc-engine-default
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.DestinationRule llm/grid-inv-e58e9806-unmatched-kserve-workload-svc
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.NetworkPolicy llm/grid-inv-e58e9806-unmatched-prometheus-scraping
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.PodMonitor llm/grid-inv-e58e9806-unmatched-kserve-llmisvc-engine
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.HTTPRoute llm/grid-inv-e58e9806-unmatched-kserve-route
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.Service llm/grid-inv-e58e9806-unmatched-kserve-workload-svc
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.Secret llm/grid-inv-e58e9806-unmatched-kserve-self-signed-certs
20:08:00	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Created	Created v1.Deployment llm/grid-inv-e58e9806-unmatched-kserve
20:08:01	llm	multus	grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
20:08:01	llm	kubelet	grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	Started	Container started
20:08:01	llm	kubelet	grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	Created	Container created
20:08:01	llm	kubelet	grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	Pulled	Container image "ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1" already present on machine and can be accessed by the pod
20:08:03 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	ProbeError	Readiness probe error: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:08:03 (x4)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-8vwd6	Unhealthy	Readiness probe failed: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:08:03	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	Updated	Updated v1.Secret llm/grid-inv-e58e9806-model-kserve-self-signed-certs
20:08:04	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	Updated	Updated v1.Secret llm/grid-inv-e58e9806-unmatched-kserve-self-signed-certs
20:08:09	llm	kubelet	grid-inv-14f915b6-unmatched-kserve-d4545864-n6ptf	Unhealthy	Readiness probe failed: Get "https://10.132.0.57:8000/ready": dial tcp 10.132.0.57:8000: connect: connection refused
20:08:09	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Unhealthy	Readiness probe failed: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:08:09	llm	LLMInferenceServiceController	grid-inv-e58e9806-model	LLMInferenceServiceReady	LLMInferenceService [grid-inv-e58e9806-model] is Ready
20:08:09	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	ProbeError	Readiness probe error: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:08:12	llm	LLMInferenceServiceController	grid-inv-e58e9806-unmatched	LLMInferenceServiceReady	LLMInferenceService [grid-inv-e58e9806-unmatched] is Ready
20:09:05 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:09:05 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	ProbeError	Readiness probe error: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:09:20 (x3)	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	ProbeError	Readiness probe error: Get "http://10.133.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
20:09:38	llm	kubelet	grid-inv-e58e9806-model-kserve-7955fd6f86-qv6r8	Killing	Stopping container main
20:09:40	llm	kubelet	grid-inv-e58e9806-unmatched-kserve-5f49c885d8-dj49d	Killing	Stopping container main
20:10:18	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-a-main-18ecf5-58568bf4599sc9g	Killing	Stopping container payload-pre-processing
20:10:18	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Killing	Stopping container payload-processing
20:10:24	openshift-ingress	kubelet	payload-processing-e2e-shared-a-main-18ecf5-699646fb48-f69hl	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.134.0.58:50052" within 1s: context deadline exceeded
20:10:25	openshift-ingress	kubelet	e2e-shared-a-main-18ecf5-openshift-default-5c586494c-4q78s	Killing	Stopping container istio-proxy
20:10:32	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Killing	Stopping container payload-processing
20:10:32	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Killing	Stopping container payload-pre-processing
20:10:38	openshift-ingress	kubelet	payload-processing-e2e-shared-b-main-dbce1a-658cd7dfdb-dn9cc	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.133.0.45:50052" within 1s: context deadline exceeded
20:10:38	openshift-ingress	kubelet	payload-pre-processing-e2e-shared-b-main-dbce1a-7dff6cd967fh44n	Unhealthy	Readiness probe failed: timeout: failed to connect service "10.133.0.46:50052" within 1s: context deadline exceeded
20:10:40	openshift-ingress	kubelet	e2e-shared-b-main-dbce1a-openshift-default-84b8d5c5cc-jm7xh	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-5xhnp
20:15:19	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 5; reason: All metrics below target
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-5xhnp	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wgkhr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-kxp6g	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-vft4f	Killing	Stopping container istio-proxy
20:15:19	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 10 to 5
20:15:19	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-wgkhr
20:15:19	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-kxp6g
20:15:19	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-88lvc
20:15:19	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-vft4f
20:15:20	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:15:20	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-88lvc	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:15:49	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-wrztq
20:15:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:15:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-f6z8p	Killing	Stopping container istio-proxy
20:15:49	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 5 to 3
20:15:49	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-f6z8p
20:15:49	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-wrztq	Killing	Stopping container istio-proxy
20:15:49	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 3; reason: All metrics below target
20:16:04	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-9jg2l
20:16:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Killing	Stopping container istio-proxy
20:16:04	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9hc8b	Killing	Stopping container istio-proxy
20:16:04	openshift-ingress	replicaset-controller	maas-default-gateway-openshift-default-67bf986f99	SuccessfulDelete	Deleted pod: maas-default-gateway-openshift-default-67bf986f99-9hc8b
20:16:04	openshift-ingress	horizontal-pod-autoscaler	maas-default-gateway-openshift-default	SuccessfulRescale	New size: 1; reason: All metrics below target
20:16:04	openshift-ingress	deployment-controller	maas-default-gateway-openshift-default	ScalingReplicaSet	Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 3 to 1
20:16:08	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	ProbeError	Readiness probe error: HTTP probe failed with statuscode: 503 body:
20:16:08	openshift-ingress	kubelet	maas-default-gateway-openshift-default-67bf986f99-9jg2l	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503